Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-04-23
13:44:17 sean-k-mooney but i dont think we can really do that for machine_type safely
13:44:48 sean-k-mooney well unless we recored the current machine type when an instance first boots
13:45:06 sean-k-mooney which would rereuie an online data migration to backfile existing instnaces
13:47:03 sean-k-mooney welll im not sure online data migration is the correct term but init_host would have to be extended to iterate over the domain on the host and populate the value in the db for all instance on the host if it is not already set
13:47:29 sean-k-mooney so we would have to start the virtdriver at a minium
13:48:14 sean-k-mooney anyway i need to bring my furniture in form my drive so ill be back in a few minutes
13:48:47 kashyap sean-k-mooney: As I said on my change, I'm very much open to the "important Plan-B"
13:48:52 kashyap artom: lyarwood: ^
13:49:10 kashyap Where "Plan-B" being [/me goes to get copy/paste text from the change ...]
13:49:38 kashyap - Have all major Nova CI jobs be 'q35' based and passing.
13:49:38 kashyap - Make installer tools set the precise q35 versioned machine types on all Compute nodes —they anyway have to do this to live migration compatibility.
13:50:52 artom sean-k-mooney, yeah, I bring this up on the spec, in upgrade impact - we're likely to "break" windows guests if we change the hardware under them
14:10:47 artom Didn't we have something more recent for "force no local disk/force boot from volume" than https://review.opendev.org/#/c/267673/?
14:20:49 lyarwood artom: we've discussed it since but I didn't think anything actually landed
14:23:25 artom lyarwood, I tracked down https://review.opendev.org/#/c/561284/
14:23:34 artom Which achieves the same thing
14:25:07 francoisp lyarwood, stephenfin, when you get a chance: https://review.opendev.org/#/c/722142/
14:25:38 lyarwood ah right I think I missed that actually landing
14:25:45 lyarwood francoisp: looking
14:26:03 stephenfin francoisp: I'm not stable core so I can't help, unfortunately :(
14:30:42 sean-k-mooney artom: we are likely to break linux guest too if they have set thing like hw_vif_model=e1000
14:30:52 sean-k-mooney or enabled any other pci only device model
14:31:07 sean-k-mooney hw_disk_bus=ide would also break
14:31:27 sean-k-mooney unless we did extra work to add an ide conttoler
14:32:43 sean-k-mooney kashyap: well in also not sure about using the versionded q35 machine type in the installeer unelss the ones that support multiple distos have a different one from each
14:32:55 sean-k-mooney for devstack we should use the unversioned one e.g. q35
14:33:01 sean-k-mooney so it just works on all distros
14:33:34 sean-k-mooney osp could use the versioned one but ooo likely will want to not as it would have to be updated with ever rhel/centos minor verions
14:37:21 kashyap sean-k-mooney: I mean whatever installer tool a given distro uses
14:37:47 kashyap sean-k-mooney: TripleO works on RPM-based distributions (CentOS, RHEL, Scientific) probably - they can set whatever they want
14:37:58 kashyap sean-k-mooney: If Ubuntu-based distros use a different installer, they will use their own
14:38:14 kashyap sean-k-mooney: For DevStack, of course, we should use unversioned - the plain alias
14:38:40 kashyap The whole point of versioned machie types is, as you might already know, to guarantee live migration compatibility
14:38:59 kashyap sean-k-mooney: TripleO upstream already does that
14:39:16 kashyap sean-k-mooney: Also, note: even for CentOS, the machine types have 'rhel' in their name.
14:39:38 lyarwood elod: could you hit https://review.opendev.org/#/c/722142/ as well please?
14:40:17 kashyap sean-k-mooney: Because CentOS obviously does not have "CentOS QEMU developers" to make the needless effort of making CentOS-based machine types. :)
14:40:34 kashyap sean-k-mooney: To summarize:
14:40:43 kashyap - DevStack -- unversioned, plain 'q35'
14:41:20 kashyap - Installers -- versioned 'q35', based on the QEMU versions your installer + Linux distro can handle
14:49:17 sean-k-mooney kashyap: i assume ooo is using the minium version supported by the os major version then in that case
14:49:38 sean-k-mooney so it will have one version that is never updated for all fo rhel/centos 8/x
14:49:47 sean-k-mooney * 8.X
14:50:45 kashyap sean-k-mooney: Why do you "minimum version"? E.g. https://review.opendev.org/#/c/716526/3/deployment/nova/nova-compute-container-puppet.yaml
14:51:22 kashyap sean-k-mooney: They update it to whatever all the Compute nodes were tested with
14:51:44 sean-k-mooney kashyap: when train released on 8.0 ooo still need to be able to install that on 8.1 or 8.2 later
14:52:04 kashyap sean-k-mooney: Sure, it will work; why would that be any problem?
14:52:15 elod lyarwood: looks good to me. +W'd
14:52:25 sean-k-mooney but if you used the max version for that majory version there would be issues with live migraion later if you scaled out a cludete ther started on 8.0 with new node on 8.1
14:52:26 kashyap If you set the machine type to 8.0; you can of course install it on 8.1 and 8.2
14:52:46 lyarwood elod: thanks
14:53:08 kashyap sean-k-mooney: Yes, the version is chosen in a way that the operator must ensure the Compute nodes are capable of it
14:53:40 kashyap sean-k-mooney: Recall, the version in the machien type matches CentOS|RHEL minor version - it is pretty clear to figure out the problem due to host OS version mismatch...
14:53:46 sean-k-mooney yep wich i ugess is the max verion supported by the oldes hypervior
14:54:02 sean-k-mooney which would be the verion form 8.0 in this case
14:54:41 kashyap Right.
14:54:46 kashyap We clear? :)
14:57:40 kashyap I take it we are.
15:02:51 sean-k-mooney kashyap: yep sorry was talking dowstream
15:03:09 sean-k-mooney and ya we are on the same page regarding the requirements here
15:03:31 kashyap No problem; thanks
15:10:21 lyarwood kashyap / sean-k-mooney ; silly question but why wouldn't we use the q35 alias all the time downstream?
15:10:42 kashyap lyarwood: Live migration compatibility
15:10:51 lyarwood kashyap: but LM wouldn't switch between the two surely?
15:10:53 kashyap lyarwood: I have an example written down, with a scenario, so you can see it plainly
15:11:04 kashyap Let me post it somewhere, so you can see it, and possibly punch holes in it
15:12:06 lyarwood ah wait so QEMU would just blindly use the alias on the dest as well?
15:12:13 lyarwood even if it pointed to a newer version
15:12:43 lyarwood sorry I assumed libvirt/QEMU would translate the alias into an actual versioned q35 machine type and stick with that for the life cycle of the domain
15:14:58 kashyap lyarwood: Yes
15:15:32 kashyap lyarwood: If you try to migrate a guest with RHEL-7.7 machine type (running on RHEL-7 COmpute node) to a RHEL-7.6 Compute node -- it fails spectacularly
15:15:45 kashyap Because 7.6 host obviously doesn't know about 7.7 :)
15:16:47 lyarwood right but my point is more that forward migration from 7.6 to 7.7 just using the q35 alias could also fail
15:17:03 lyarwood as the domain would jump from the 7.6 to 7.7 version of the machine type
15:17:05 kashyap lyarwood: https://kashyapc.fedorapeople.org/versioned-machine-types-and-live-migration-gotcha.txt
15:17:08 lyarwood well maybe not fail but things could change
15:17:27 kashyap lyarwood: Forward migration fomr 7.6 to 7.7 will _not_ fail
15:17:28 kashyap Why?
15:17:55 kashyap Because, 7.7 host will have all the older (7.6, 7.5, 7.4) machine types precisely for live migration compat
15:18:07 kashyap lyarwood: I had the above text somewhere in the spec, but deleted it, for "brevity"
15:18:23 kashyap artom: Since you also spent non-trivial time reviewing the spec, you might benefit from reading this: https://kashyapc.fedorapeople.org/versioned-machine-types-and-live-migration-gotcha.txt
15:18:42 lyarwood It will now be _impossible_ to migrate a guest from a RHEL 7.6-based
15:18:43 lyarwood compute node to a RHEL 7.7-based — because RHEL 7.6 won't know about
15:18:45 lyarwood the RHEL 7.7's machine type.
15:18:56 lyarwood kashyap: ^ that was my point sorry
15:19:25 lyarwood and the workaround appears to be to pin to the oldest version
15:19:33 kashyap Yep
15:19:40 lyarwood but I just don't get how this works downstream across multple releases
15:19:49 lyarwood we'd basically need to pin to the first version used in the env
15:20:55 openstackgerrit Takashi Natsume proposed openstack/nova master: Fix list rendering in the accelerator support doc https://review.opendev.org/721846
15:21:58 lyarwood surely we should work out and persist the versioned machine type during a migration?
15:22:24 lyarwood so if the source is 8.1 and using the q35 alias ensure it uses that on the dest
15:22:36 lyarwood but all new instances use the 8.2 q35 type?
15:22:56 kashyap lyarwood: For downstream, yes: you need to pick a versioned machine type and set it across the nodes before deplyoying.
15:22:59 aarents lyarwood: kashyap we hit this kind of issues.. but Im not the guy who work on it I will check how we dealt with that..
15:23:01 lyarwood kashyap: sorry I'm talking out loud here, could you add the LM section back into the spec and I'll add words there.
15:23:26 kashyap lyarwood: Sure, let me add that bit in the spec in a separate section; it's important
15:23:30 lyarwood kashyap: yeah the issue with that is you can never move from it
15:24:16 kashyap lyarwood: Hang on - you _can_ move; it's a step-by-step process:
15:25:35 kashyap lyarwood: Err, my "Here's the problem" has a bug!

Earlier   Later