| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-03 | |||
| 18:40:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add volume-backed evacuate test https://review.openstack.org/604397 | |
| 18:41:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Run evacuate tests with local/lvm and shared/rbd storage https://review.openstack.org/604400 | |
| 18:41:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix InstanceNotFound during _destroy_evacuated_instances https://review.openstack.org/606122 | |
| 18:41:00 | openstack | bug 1794996 in OpenStack Compute (nova) "_destroy_evacuated_instances fails and kills n-cpu startup if lazy-loading flavor on a deleted instance" [High,In progress] https://launchpad.net/bugs/1794996 - Assigned to Matt Riedemann (mriedem) | |
| 18:41:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for bug 1794996 https://review.openstack.org/606106 | |
| 18:41:26 | spatel | sean-k-mooney: should i use this? hugepagesz=2M hugepages=15360 | |
| 18:41:41 | spatel | it will give 30G | |
| 18:41:56 | spatel | let me try to make it 28G | |
| 18:42:03 | spatel | keep 4G for OS | |
| 18:42:27 | sean-k-mooney | the hugepage memory will not be availabel to normal os process so 2MB is likely too tight for a compute node | |
| 18:43:28 | sean-k-mooney | 4G should be ok but used to give ^G as my safty margin that said i did not need that much of a margin | |
| 18:44:17 | spatel | In that case let me give 8G to OS (keep 24G for VM) | |
| 18:44:32 | sean-k-mooney | spatel: i would set it to 12 288 | |
| 18:44:37 | sean-k-mooney | which is 24G | |
| 18:45:03 | spatel | hugepagesz=2M hugepages=12288 - DONE! going to reboot compute node | |
| 18:45:31 | spatel | Do you use isolcpus= CPUAffinity ? | |
| 18:45:47 | spatel | I was reading about that not sure i need to worry about that or not | |
| 18:46:03 | sean-k-mooney | i would then also reduce the max size vms to 10 or 12 GB ram for your largset flavor so you can alway boot at least 2 of them | |
| 18:46:29 | sean-k-mooney | isolcpus is not the same as cpuaffintiy | |
| 18:47:00 | sean-k-mooney | i generally avoid isolcpus= it is a rather large hammer to reach for | |
| 18:47:16 | mriedem | bauzas: i've -2ed https://review.openstack.org/#/c/599208/ as we discussed yesterday | |
| 18:47:17 | sean-k-mooney | it should only be used for realtime instances even then its tricky to use correctly | |
| 18:47:34 | spatel | ok! got it | |
| 18:47:45 | sean-k-mooney | spatel: generally i would only suggest usign it to isolage cores allcoated to ovs-dpdk if you chose to depoly it | |
| 18:48:53 | sean-k-mooney | spatel: dont get me wronge isolcpus= has a place but its only somting i reach for when i have no other options left and i really really need it | |
| 18:49:07 | spatel | I will soon deploy dpdk (believe me) | |
| 18:49:57 | mnaser | sean-k-mooney: https://review.openstack.org/#/c/607686/ .. ill push up a patch to test things out when possible (or at least something to confirm its working) | |
| 18:50:01 | spatel | in flavor i should set hw:mem_page_size='2048' right ? | |
| 18:50:07 | mnaser | so maybe if you want to start figuring out nova dependencies | |
| 18:52:41 | dansmith | mriedem: melwitt tssurya: cells meeting today? I have an appointment the hour before, but I will probably be back in time | |
| 18:52:51 | sean-k-mooney | spatel: you can but i prefer seting hw:mem_page_size=large | |
| 18:53:05 | sean-k-mooney | spatel: that will work with both 1G and 2MB hugepages | |
| 18:53:15 | spatel | done! let me do that | |
| 18:53:21 | dansmith | side note, mriedem melwitt: This is easy early utility stuff we can merge in front of the down cell stuff: https://review.openstack.org/#/c/594947/ | |
| 18:54:01 | tssurya | dansmith: the most important question I had was the best way to get the "type" of exception from the utility ^ | |
| 18:54:22 | dansmith | type? | |
| 18:54:25 | tssurya | we could also do it during the meeting if others also have topics | |
| 18:54:43 | mriedem | dansmith: i was holding off on that one until i knew what was going on further in the series | |
| 18:54:52 | tssurya | yea for instance a TimeOut/DBonnectionError expception versus InstanceNotFound exception | |
| 18:55:09 | tssurya | as of now we always return the "raised_exception_sentinel" which is not that useful | |
| 18:55:27 | tssurya | because based on the type of exception we have to handle it differently | |
| 18:55:31 | nicolasbock | Fixing the migration is more difficult it seems: I successfully updated the DB with the correct hypervisor and `server show` was now showing the correct hypervisor information | |
| 18:55:37 | mriedem | please hold, i have to sell something to a craigslist weirdo real quick | |
| 18:55:38 | dansmith | tssurya: by timeout you mean an rpc timeout, not the did_not_respond_sentinel I assume? | |
| 18:55:46 | nicolasbock | I ran `server migrate` which failed with `[Errno 2] No such file or directory: '/var/lib/nova/instances/2aa3a324-bf22-4e0c-912a-d7c52f59f1fd/disk` | |
| 18:55:58 | nicolasbock | So the disk didn't make it in the first migration | |
| 18:56:04 | nicolasbock | I verified that the disk is still on the old host | |
| 18:56:18 | sean-k-mooney | mnaser: that spec would allow testing quite alot of featue espcially if it supproted nested virt | |
| 18:56:29 | nicolasbock | Since I am in the middle of open heart surgery anyway I figured I just rsync the disk to the current hypervisor | |
| 18:56:35 | tssurya | dansmith: TimeOut was just an example, my main problem is to filter the "InstanceNotFOund" from others for nvoa show | |
| 18:56:37 | tssurya | nvoa show* | |
| 18:56:41 | nicolasbock | So that worked | |
| 18:56:41 | tssurya | nova show* | |
| 18:56:44 | mnaser | sean-k-mooney: these vms have nested virt | |
| 18:56:57 | nicolasbock | However, migrate is now refusing to migrate since the VM is in an ERROR state | |
| 18:57:08 | dansmith | tssurya: yeah | |
| 18:57:15 | mnaser | nicolasbock: nova reset-state --active | |
| 18:57:16 | nicolasbock | I can't `nova reset-state` either, it says `Reset state for server 2aa3a324-bf22-4e0c-912a-d7c52f59f1fd succeeded; new state is error` | |
| 18:57:22 | tssurya | as of now, when we get the InstanceNotFound, utility hides this returns the sentinel, I try to go and make a minimal construct when I shouldn't be | |
| 18:57:23 | nicolasbock | which isn't all that helpful :( | |
| 18:57:30 | mnaser | `--active` | |
| 18:57:30 | dansmith | tssurya: probably have to get away from the sentinel object I guess | |
| 18:57:38 | dansmith | tssurya: which is going to be a mess | |
| 18:57:41 | tssurya | melwitt and I had a brief discussion | |
| 18:57:53 | melwitt | dansmith, tssurya: sean-k-mooney proposed this class as a way to be able to return exception objects https://review.openstack.org/605251 | |
| 18:57:56 | tssurya | the other day | |
| 18:57:57 | nicolasbock | Yeah mnaser !!! | |
| 18:58:25 | nicolasbock | I hadn't considered that since `--active Request the server be reset to "active" state instead of "error" state (the default).` | |
| 18:58:34 | dansmith | um | |
| 18:58:54 | nicolasbock | I guess `--active` isn't the default after all | |
| 18:59:05 | dansmith | seems a lot overkill :) | |
| 18:59:08 | sean-k-mooney | mnaser: do you provide any other custom nodes. i dont know if you care about ovs-dpdk or cpu pinnng but would you be ok if we used that or a sligly less different flavor to maybe test does feature in the gate? | |
| 18:59:41 | mriedem | nicolasbock: that disk not found with cold migration sounds like a bug i've seen before that is fixed, but had to do with shared storage and volume-backed instances | |
| 18:59:41 | dansmith | melwitt: tssurya: it would be trivial to just use the exception as the sentinel in the response, and we just check to see if the result isinstance(thing, Exception) | |
| 18:59:44 | melwitt | dansmith: comment on the review :) it came about because I said something like, can we return the exception object in addition to the sentinel, in a tuple or something | |
| 18:59:45 | dansmith | and then you have the exception itself | |
| 18:59:49 | mnaser | we are slowly rolling out nested virt across our entire fleet but that is something to discuss more with the infra team i think | |
| 18:59:52 | mriedem | nicolasbock: but likely not fixed on newton | |
| 19:00:23 | melwitt | dansmith: yeah, that was my other suggestion. I had two ideas: drop the sentinel and check isinstance or keep the sentinel and have tuples | |
| 19:00:26 | tssurya | dansmith: right, that would be simple, is it okay to change the utility's face now ? | |
| 19:00:30 | nicolasbock | ok, do you happen to remember the review this was fixed in mriedem ? Maybe I can backport? | |
| 19:00:38 | mriedem | looking | |
| 19:00:47 | dansmith | melwitt: no reason for the sentinel I don't think | |
| 19:01:01 | nicolasbock | Thanks mriedem | |
| 19:01:04 | dansmith | anything that isinstance(Exception) is... an error, so... | |
| 19:01:15 | nicolasbock | mnaser: it worked! The VM has migrated to a new host | |
| 19:01:27 | melwitt | dansmith: yeah, that's what I was thinking | |
| 19:01:32 | sean-k-mooney | mnaser: for ovs-dpdk and cpu pinning/hugepages we dont need nvme or gpus but we do need nested virt and a vm with multiple numa nodes. it is somthing that i agree i would love to discuss with infra. | |
| 19:01:44 | mnaser | yeah, we'd have to talk it out with infra | |
| 19:01:56 | melwitt | dansmith: but sean-k-mooney was thinking checking isinstance was an anti-pattern of some kind | |
| 19:02:14 | sean-k-mooney | melwitt: sorry i should read the scrole back | |
| 19:02:30 | melwitt | sean-k-mooney: we're just talking about the "return exceptions from scatter-gather" thing | |
| 19:02:31 | dansmith | melwitt: overengineering is an anti-pattern :) | |
| 19:02:46 | tssurya | sean-k-mooney: its about this: https://review.openstack.org/#/c/605251/ | |
| 19:03:05 | mriedem | nicolasbock: https://review.openstack.org/#/q/Ib10081150e125961cba19cfa821bddfac4614408 is what i'm thinking of | |
| 19:03:19 | nicolasbock | Is it ok that the disk is still on the old host after migration? | |
| 19:03:20 | melwitt | sean-k-mooney: dansmith suggested the same thing I suggested when we first talked about it, just return exception objects instead of the sentinel and check isinstance(thing, Exception) to know whether an error was returned or not | |
| 19:03:22 | nicolasbock | Thanks mriedem | |
| 19:03:25 | sean-k-mooney | dansmith: well i was porting a standard calass form c++ to python. retrun an exception has some weird sidefect in python 2 | |
| 19:04:16 | dansmith | sean-k-mooney: I have no idea what weird side effect you mean, other than that re-raising it doesn't keep the exception context properly, but we won't be doing that here | |