| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-23 | |||
| 13:43:28 | sean-k-mooney | so you can opt in to different defaults but not actully change them | |
| 13:43:52 | sean-k-mooney | my understanding is kashyap was advocating for changing the default in the libvirt driver directly | |
| 13:44:01 | sean-k-mooney | kind of like we did for the random number generator | |
| 13:44:17 | sean-k-mooney | but i dont think we can really do that for machine_type safely | |
| 13:44:48 | sean-k-mooney | well unless we recored the current machine type when an instance first boots | |
| 13:45:06 | sean-k-mooney | which would rereuie an online data migration to backfile existing instnaces | |
| 13:47:03 | sean-k-mooney | welll im not sure online data migration is the correct term but init_host would have to be extended to iterate over the domain on the host and populate the value in the db for all instance on the host if it is not already set | |
| 13:47:29 | sean-k-mooney | so we would have to start the virtdriver at a minium | |
| 13:48:14 | sean-k-mooney | anyway i need to bring my furniture in form my drive so ill be back in a few minutes | |
| 13:48:47 | kashyap | sean-k-mooney: As I said on my change, I'm very much open to the "important Plan-B" | |
| 13:48:52 | kashyap | artom: lyarwood: ^ | |
| 13:49:10 | kashyap | Where "Plan-B" being [/me goes to get copy/paste text from the change ...] | |
| 13:49:38 | kashyap | - Have all major Nova CI jobs be 'q35' based and passing. | |
| 13:49:38 | kashyap | - Make installer tools set the precise q35 versioned machine types on all Compute nodes —they anyway have to do this to live migration compatibility. | |
| 13:50:52 | artom | sean-k-mooney, yeah, I bring this up on the spec, in upgrade impact - we're likely to "break" windows guests if we change the hardware under them | |
| 14:10:47 | artom | Didn't we have something more recent for "force no local disk/force boot from volume" than https://review.opendev.org/#/c/267673/? | |
| 14:20:49 | lyarwood | artom: we've discussed it since but I didn't think anything actually landed | |
| 14:23:25 | artom | lyarwood, I tracked down https://review.opendev.org/#/c/561284/ | |
| 14:23:34 | artom | Which achieves the same thing | |
| 14:25:07 | francoisp | lyarwood, stephenfin, when you get a chance: https://review.opendev.org/#/c/722142/ | |
| 14:25:38 | lyarwood | ah right I think I missed that actually landing | |
| 14:25:45 | lyarwood | francoisp: looking | |
| 14:26:03 | stephenfin | francoisp: I'm not stable core so I can't help, unfortunately :( | |
| 14:30:42 | sean-k-mooney | artom: we are likely to break linux guest too if they have set thing like hw_vif_model=e1000 | |
| 14:30:52 | sean-k-mooney | or enabled any other pci only device model | |
| 14:31:07 | sean-k-mooney | hw_disk_bus=ide would also break | |
| 14:31:27 | sean-k-mooney | unless we did extra work to add an ide conttoler | |
| 14:32:43 | sean-k-mooney | kashyap: well in also not sure about using the versionded q35 machine type in the installeer unelss the ones that support multiple distos have a different one from each | |
| 14:32:55 | sean-k-mooney | for devstack we should use the unversioned one e.g. q35 | |
| 14:33:01 | sean-k-mooney | so it just works on all distros | |
| 14:33:34 | sean-k-mooney | osp could use the versioned one but ooo likely will want to not as it would have to be updated with ever rhel/centos minor verions | |
| 14:37:21 | kashyap | sean-k-mooney: I mean whatever installer tool a given distro uses | |
| 14:37:47 | kashyap | sean-k-mooney: TripleO works on RPM-based distributions (CentOS, RHEL, Scientific) probably - they can set whatever they want | |
| 14:37:58 | kashyap | sean-k-mooney: If Ubuntu-based distros use a different installer, they will use their own | |
| 14:38:14 | kashyap | sean-k-mooney: For DevStack, of course, we should use unversioned - the plain alias | |
| 14:38:40 | kashyap | The whole point of versioned machie types is, as you might already know, to guarantee live migration compatibility | |
| 14:38:59 | kashyap | sean-k-mooney: TripleO upstream already does that | |
| 14:39:16 | kashyap | sean-k-mooney: Also, note: even for CentOS, the machine types have 'rhel' in their name. | |
| 14:39:38 | lyarwood | elod: could you hit https://review.opendev.org/#/c/722142/ as well please? | |
| 14:40:17 | kashyap | sean-k-mooney: Because CentOS obviously does not have "CentOS QEMU developers" to make the needless effort of making CentOS-based machine types. :) | |
| 14:40:34 | kashyap | sean-k-mooney: To summarize: | |
| 14:40:43 | kashyap | - DevStack -- unversioned, plain 'q35' | |
| 14:41:20 | kashyap | - Installers -- versioned 'q35', based on the QEMU versions your installer + Linux distro can handle | |
| 14:49:17 | sean-k-mooney | kashyap: i assume ooo is using the minium version supported by the os major version then in that case | |
| 14:49:38 | sean-k-mooney | so it will have one version that is never updated for all fo rhel/centos 8/x | |
| 14:49:47 | sean-k-mooney | * 8.X | |
| 14:50:45 | kashyap | sean-k-mooney: Why do you "minimum version"? E.g. https://review.opendev.org/#/c/716526/3/deployment/nova/nova-compute-container-puppet.yaml | |
| 14:51:22 | kashyap | sean-k-mooney: They update it to whatever all the Compute nodes were tested with | |
| 14:51:44 | sean-k-mooney | kashyap: when train released on 8.0 ooo still need to be able to install that on 8.1 or 8.2 later | |
| 14:52:04 | kashyap | sean-k-mooney: Sure, it will work; why would that be any problem? | |
| 14:52:15 | elod | lyarwood: looks good to me. +W'd | |
| 14:52:25 | sean-k-mooney | but if you used the max version for that majory version there would be issues with live migraion later if you scaled out a cludete ther started on 8.0 with new node on 8.1 | |
| 14:52:26 | kashyap | If you set the machine type to 8.0; you can of course install it on 8.1 and 8.2 | |
| 14:52:46 | lyarwood | elod: thanks | |
| 14:53:08 | kashyap | sean-k-mooney: Yes, the version is chosen in a way that the operator must ensure the Compute nodes are capable of it | |
| 14:53:40 | kashyap | sean-k-mooney: Recall, the version in the machien type matches CentOS|RHEL minor version - it is pretty clear to figure out the problem due to host OS version mismatch... | |
| 14:53:46 | sean-k-mooney | yep wich i ugess is the max verion supported by the oldes hypervior | |
| 14:54:02 | sean-k-mooney | which would be the verion form 8.0 in this case | |
| 14:54:41 | kashyap | Right. | |
| 14:54:46 | kashyap | We clear? :) | |
| 14:57:40 | kashyap | I take it we are. | |
| 15:02:51 | sean-k-mooney | kashyap: yep sorry was talking dowstream | |
| 15:03:09 | sean-k-mooney | and ya we are on the same page regarding the requirements here | |
| 15:03:31 | kashyap | No problem; thanks | |
| 15:10:21 | lyarwood | kashyap / sean-k-mooney ; silly question but why wouldn't we use the q35 alias all the time downstream? | |
| 15:10:42 | kashyap | lyarwood: Live migration compatibility | |
| 15:10:51 | lyarwood | kashyap: but LM wouldn't switch between the two surely? | |
| 15:10:53 | kashyap | lyarwood: I have an example written down, with a scenario, so you can see it plainly | |
| 15:11:04 | kashyap | Let me post it somewhere, so you can see it, and possibly punch holes in it | |
| 15:12:06 | lyarwood | ah wait so QEMU would just blindly use the alias on the dest as well? | |
| 15:12:13 | lyarwood | even if it pointed to a newer version | |
| 15:12:43 | lyarwood | sorry I assumed libvirt/QEMU would translate the alias into an actual versioned q35 machine type and stick with that for the life cycle of the domain | |
| 15:14:58 | kashyap | lyarwood: Yes | |
| 15:15:32 | kashyap | lyarwood: If you try to migrate a guest with RHEL-7.7 machine type (running on RHEL-7 COmpute node) to a RHEL-7.6 Compute node -- it fails spectacularly | |
| 15:15:45 | kashyap | Because 7.6 host obviously doesn't know about 7.7 :) | |
| 15:16:47 | lyarwood | right but my point is more that forward migration from 7.6 to 7.7 just using the q35 alias could also fail | |
| 15:17:03 | lyarwood | as the domain would jump from the 7.6 to 7.7 version of the machine type | |
| 15:17:05 | kashyap | lyarwood: https://kashyapc.fedorapeople.org/versioned-machine-types-and-live-migration-gotcha.txt | |
| 15:17:08 | lyarwood | well maybe not fail but things could change | |
| 15:17:27 | kashyap | lyarwood: Forward migration fomr 7.6 to 7.7 will _not_ fail | |
| 15:17:28 | kashyap | Why? | |
| 15:17:55 | kashyap | Because, 7.7 host will have all the older (7.6, 7.5, 7.4) machine types precisely for live migration compat | |
| 15:18:07 | kashyap | lyarwood: I had the above text somewhere in the spec, but deleted it, for "brevity" | |
| 15:18:23 | kashyap | artom: Since you also spent non-trivial time reviewing the spec, you might benefit from reading this: https://kashyapc.fedorapeople.org/versioned-machine-types-and-live-migration-gotcha.txt | |
| 15:18:42 | lyarwood | It will now be _impossible_ to migrate a guest from a RHEL 7.6-based | |
| 15:18:43 | lyarwood | compute node to a RHEL 7.7-based — because RHEL 7.6 won't know about | |
| 15:18:45 | lyarwood | the RHEL 7.7's machine type. | |
| 15:18:56 | lyarwood | kashyap: ^ that was my point sorry | |
| 15:19:25 | lyarwood | and the workaround appears to be to pin to the oldest version | |
| 15:19:33 | kashyap | Yep | |
| 15:19:40 | lyarwood | but I just don't get how this works downstream across multple releases | |
| 15:19:49 | lyarwood | we'd basically need to pin to the first version used in the env | |
| 15:20:55 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Fix list rendering in the accelerator support doc https://review.opendev.org/721846 | |
| 15:21:58 | lyarwood | surely we should work out and persist the versioned machine type during a migration? | |
| 15:22:24 | lyarwood | so if the source is 8.1 and using the q35 alias ensure it uses that on the dest | |
| 15:22:36 | lyarwood | but all new instances use the 8.2 q35 type? | |
| 15:22:56 | kashyap | lyarwood: For downstream, yes: you need to pick a versioned machine type and set it across the nodes before deplyoying. | |
| 15:22:59 | aarents | lyarwood: kashyap we hit this kind of issues.. but Im not the guy who work on it I will check how we dealt with that.. | |
| 15:23:01 | lyarwood | kashyap: sorry I'm talking out loud here, could you add the LM section back into the spec and I'll add words there. | |
| 15:23:26 | kashyap | lyarwood: Sure, let me add that bit in the spec in a separate section; it's important | |