| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-07 | |||
| 17:12:38 | idlemind | sean-k-mooney would that be a bug in centos (my base kvm node for openstack) or in nova if all the cpu instructions aren't getting passed into the guest | |
| 17:12:38 | hrw | stephenfin: I like that fact - easier to use same tools despite hypervisor differences | |
| 17:12:50 | mriedem | yeah so guest.os_mach_type is per image, | |
| 17:12:55 | mriedem | so doesn't belong in _do_quality_warnings | |
| 17:13:16 | stephenfin | mriedem: Yup, just replied. The warning would have to go in that same area of the code | |
| 17:13:17 | mriedem | otherwise based on the host arch | |
| 17:13:20 | sean-k-mooney | idlemind: i think a centos/libvirt/qemu/kvm bug | |
| 17:13:52 | sean-k-mooney | idlemind: how did you enable nested virt in kvm? what module arg did you set it is different on amd vs intel | |
| 17:13:52 | idlemind | sean-k-mooney k thx i can start there; any chance you know which metadata property you were thinking of to add the force for me as a work-around? | |
| 17:14:47 | mriedem | wow even danpb was reviewing this patch | |
| 17:14:54 | idlemind | sean-k-mooney for centos in /etc/modprobe.d/kvm.conf they have the necessary lines for amd and intel you just uncomment the one you need. you can then reload the kvm_amd (in my case) module or reboot the host. i did reboots on both my hosts. | |
| 17:15:50 | idlemind | sean-k-mooney i set nova.conf overrides to set cpu_mode = passthrough via openstack ansible i did not do anything with cpu_model | |
| 17:15:50 | bkero- | echo "options kvm_intel nested=1" > /etc/modprobe.d/kvm.conf # or kvm_amd if you have it, then... modprobe -r kvm_intel; modprobe kvm_intel | |
| 17:16:22 | sean-k-mooney | idlemind: ah its not passthrough its host-passthrough i belive | |
| 17:17:02 | sean-k-mooney | idlemind: yes it should be "host-passthrough" | |
| 17:17:09 | sean-k-mooney | see https://docs.openstack.org/nova/queens/configuration/config.html | |
| 17:17:09 | hrw | mriedem, stephenfin: replied | |
| 17:17:13 | idlemind | i think that's what i use because that comes through fine | |
| 17:17:36 | hrw | mriedem: I asked libvirt guys to take a look to make sure that I got the idea properly | |
| 17:18:14 | hrw | mriedem: as we were discussing (on irc) few ways of handling it and only this one turned out to be proper working without any random glitches | |
| 17:18:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Rename the 'recreate' param in rebuild_instance to 'evacuate' https://review.openstack.org/508190 | |
| 17:19:02 | cdent | well that's not confusing at all | |
| 17:19:04 | idlemind | sean-k-mooney and bkero- http://paste.openstack.org/show/693686/ | |
| 17:19:42 | idlemind | w/o that feature policy='require' name='svm' i don't get svm in my guest and kvm-ok fails | |
| 17:20:00 | mriedem | cdent: you weren't in the room on friday | |
| 17:20:05 | mriedem | when my head exploded | |
| 17:20:25 | cdent | that was the Nth time where N is ? | |
| 17:20:27 | sean-k-mooney | idlemind: and if you do a cat /proc/cpuinfo on the host you see svm listed | |
| 17:20:48 | hrw | stephenfin: thanks for +2 ;) now have to fetch another one and +w ;D | |
| 17:20:49 | idlemind | sean-k-mooney yes w/the feature line i get svm in /proc/cpuinfo ... w/o the feature line no dice | |
| 17:21:08 | sean-k-mooney | idlemind: no i mean on the host not the guest | |
| 17:21:24 | mriedem | bauzas: dansmith: given the rebuild hoopla, and cve regression circus, we should probably get this in and backport as well https://review.openstack.org/#/c/546268/ | |
| 17:21:42 | mriedem | while i'm thinking about rebuild | |
| 17:21:55 | idlemind | sean-k-mooney yes for all 4 cores | |
| 17:21:57 | mriedem | hrw: i've got a tab open, will look in detail later | |
| 17:22:01 | hrw | mriedem: thanks | |
| 17:22:41 | sean-k-mooney | idlemind: strange so ya i think this is a libvirt/kvm/qemu bug and it it is not correctly reading the cpu flags | |
| 17:23:23 | sean-k-mooney | idlemind: i assume you are using qemu/libvirt for the rdo repos not the default centos one? | |
| 17:24:17 | idlemind | sean-k-mooney i'm not sure tbh i'm thinking they would have been brought in by openstack-ansible so i'd have to dig around there | |
| 17:24:52 | idlemind | http://paste.openstack.org/show/693687/ | |
| 17:25:20 | idlemind | looks like a fairly new release | |
| 17:25:23 | sean-k-mooney | the -ev rpms are form rdo | |
| 17:25:42 | sean-k-mooney | the default centos ones are old and busted in many ways | |
| 17:26:46 | dansmith | mriedem: I don't understand the logic in the comment you're removing | |
| 17:26:50 | openstackgerrit | Merged openstack/nova master: Fix version cap when no nova-compute started https://review.openstack.org/549737 | |
| 17:27:06 | dansmith | mriedem: seems like cases 2 and 3 are both covered by recreate=True, why was it even considering the scheduled_node in there? | |
| 17:28:38 | idlemind | sean-k-mooney well at least that's a good sign! | |
| 17:28:43 | mriedem | dansmith: honestly i think the comment was just overthinking things when it was written to fix a similar type of issue with forced host evacuate | |
| 17:28:59 | mriedem | that's why i removed it and just changed the code to check the recreate flag | |
| 17:29:31 | mriedem | the scheduled_node thing was tied into whether or not you were doing a forced host evacuate, | |
| 17:29:43 | mriedem | and the comment was added at the same time as fixing a bug in that scenario, so it was probably just 'on the brain' | |
| 17:30:21 | mriedem | that comment also predates the changes to run the host back through the scheduler on rebuild if the image changes | |
| 17:30:39 | dansmith | mriedem: yeah I just want to make sure I understand what it was trying to do | |
| 17:30:49 | mriedem | anyway, all related to said head explosion at the ptg | |
| 17:30:58 | dansmith | mriedem: looking at the original patch, "if scheduled_node is not None" which is was already there, | |
| 17:31:04 | dansmith | so it was probably just trying to augment that case, | |
| 17:31:14 | dansmith | when really it should have just considered the recreate flag itself | |
| 17:31:46 | mriedem | i'd have to go back and look at that patch to get it all back into context | |
| 17:32:10 | mriedem | i think it boiled down to, forcing a host on evacuate never worked since it was added in newton until that patch fixed it :) | |
| 17:32:30 | mriedem | was also somehow related to pci devices, so maybe it was that it never claimed? can't remember | |
| 17:32:46 | mriedem | the claim is what messes with the pci devices in the migration context right? | |
| 17:32:53 | dansmith | idk | |
| 17:32:57 | mriedem | heh same here | |
| 17:38:17 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Only attempt a rebuild claim for an evacuation to a new host https://review.openstack.org/550545 | |
| 17:41:33 | sean-k-mooney | mriedem: there are two things that mess with pci device for migration. one we dont update allcoation correctly and 2 we dont regenerate the xml for that pci passhtrough section. | |
| 17:42:14 | sean-k-mooney | mriedem: that not a new thing though livemigration with pci devices never worked | |
| 17:42:45 | mriedem | artom is going to fix it | |
| 17:42:49 | mriedem | or die trying | |
| 17:43:09 | mriedem | sean-k-mooney: what do you mean by "don't update allocation correctly"? | |
| 17:43:12 | mriedem | claims? | |
| 17:43:22 | sean-k-mooney | mriedem: yes | |
| 17:43:23 | mriedem | or something in the pci tracker? | |
| 17:43:26 | mriedem | that's all voodoo to me | |
| 17:43:34 | sean-k-mooney | well the pci device table in the db | |
| 17:43:53 | sean-k-mooney | when we migrate we dont claim the device on the new node if i remeber correctly | |
| 17:44:10 | sean-k-mooney | and i dont think we release it on the old node either | |
| 17:45:34 | sean-k-mooney | if someone does fix that then awsome but i basically equate pci passthough = no migration even though we can make it work if we really want | |
| 17:46:26 | mriedem | maybe i'm thinking about pinned cpus | |
| 17:46:28 | mriedem | L226 https://etherpad.openstack.org/p/nova-ptg-rocky-placement | |
| 17:47:05 | sean-k-mooney | pinned cpus has a similar issue in that we dont regenerate the xml to account for the available cpus on the destination | |
| 17:47:29 | mriedem | right i think that's what artom is going to spec up and fix | |
| 17:47:44 | sean-k-mooney | mriedem: artom or stephenfin ? | |
| 17:47:47 | mriedem | artom | |
| 17:48:25 | mriedem | i guess live migrating instances with pci devices will wait for nested RPs (or something) | |
| 17:48:37 | sean-k-mooney | that would be nice to fix too. i think cold migrate works today but just not live migrate | |
| 17:49:18 | sean-k-mooney | mriedem: ya or after the multiple port binding work is done perhaps. | |
| 17:49:49 | sean-k-mooney | i think cold migrate might work with pci passhtouhg though not sure. | |
| 17:49:51 | mriedem | the cpu pinning spec is going to follow the port binding template | |
| 17:50:32 | sean-k-mooney | mriedem: for the fallback and version check in the conductor to ensure both nodes support it | |
| 17:50:45 | mriedem | yes something like that | |
| 17:50:55 | mriedem | or that the requested cpus to pin are available on the dest | |
| 17:50:58 | mriedem | and/or | |
| 17:51:05 | mriedem | if not, fail that host and try another one | |
| 17:51:51 | sean-k-mooney | ya that makes sense however with the other placement discusstion about modeling dedicated cpus as a new inventory placement can gurantee that for us | |
| 17:52:15 | mriedem | is there anything placement *can't* do? | |
| 17:52:29 | mriedem | i wish placement would have my babies | |
| 17:52:30 | sean-k-mooney | well we will still need the numa toplology filter i guest but we should be able to confirm the cpus are free before calling the compute node | |
| 17:53:39 | openstackgerrit | Merged openstack/nova master: Handle not found error on taking snapshot https://review.openstack.org/511074 | |
| 18:06:10 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: Snapshot https://review.openstack.org/543023 | |
| 18:12:38 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Only attempt a rebuild claim for an evacuation to a new host https://review.openstack.org/550555 | |