| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-23 | |||
| 14:30:52 | sean-k-mooney | or enabled any other pci only device model | |
| 14:31:07 | sean-k-mooney | hw_disk_bus=ide would also break | |
| 14:31:27 | sean-k-mooney | unless we did extra work to add an ide conttoler | |
| 14:32:43 | sean-k-mooney | kashyap: well in also not sure about using the versionded q35 machine type in the installeer unelss the ones that support multiple distos have a different one from each | |
| 14:32:55 | sean-k-mooney | for devstack we should use the unversioned one e.g. q35 | |
| 14:33:01 | sean-k-mooney | so it just works on all distros | |
| 14:33:34 | sean-k-mooney | osp could use the versioned one but ooo likely will want to not as it would have to be updated with ever rhel/centos minor verions | |
| 14:37:21 | kashyap | sean-k-mooney: I mean whatever installer tool a given distro uses | |
| 14:37:47 | kashyap | sean-k-mooney: TripleO works on RPM-based distributions (CentOS, RHEL, Scientific) probably - they can set whatever they want | |
| 14:37:58 | kashyap | sean-k-mooney: If Ubuntu-based distros use a different installer, they will use their own | |
| 14:38:14 | kashyap | sean-k-mooney: For DevStack, of course, we should use unversioned - the plain alias | |
| 14:38:40 | kashyap | The whole point of versioned machie types is, as you might already know, to guarantee live migration compatibility | |
| 14:38:59 | kashyap | sean-k-mooney: TripleO upstream already does that | |
| 14:39:16 | kashyap | sean-k-mooney: Also, note: even for CentOS, the machine types have 'rhel' in their name. | |
| 14:39:38 | lyarwood | elod: could you hit https://review.opendev.org/#/c/722142/ as well please? | |
| 14:40:17 | kashyap | sean-k-mooney: Because CentOS obviously does not have "CentOS QEMU developers" to make the needless effort of making CentOS-based machine types. :) | |
| 14:40:34 | kashyap | sean-k-mooney: To summarize: | |
| 14:40:43 | kashyap | - DevStack -- unversioned, plain 'q35' | |
| 14:41:20 | kashyap | - Installers -- versioned 'q35', based on the QEMU versions your installer + Linux distro can handle | |
| 14:49:17 | sean-k-mooney | kashyap: i assume ooo is using the minium version supported by the os major version then in that case | |
| 14:49:38 | sean-k-mooney | so it will have one version that is never updated for all fo rhel/centos 8/x | |
| 14:49:47 | sean-k-mooney | * 8.X | |
| 14:50:45 | kashyap | sean-k-mooney: Why do you "minimum version"? E.g. https://review.opendev.org/#/c/716526/3/deployment/nova/nova-compute-container-puppet.yaml | |
| 14:51:22 | kashyap | sean-k-mooney: They update it to whatever all the Compute nodes were tested with | |
| 14:51:44 | sean-k-mooney | kashyap: when train released on 8.0 ooo still need to be able to install that on 8.1 or 8.2 later | |
| 14:52:04 | kashyap | sean-k-mooney: Sure, it will work; why would that be any problem? | |
| 14:52:15 | elod | lyarwood: looks good to me. +W'd | |
| 14:52:25 | sean-k-mooney | but if you used the max version for that majory version there would be issues with live migraion later if you scaled out a cludete ther started on 8.0 with new node on 8.1 | |
| 14:52:26 | kashyap | If you set the machine type to 8.0; you can of course install it on 8.1 and 8.2 | |
| 14:52:46 | lyarwood | elod: thanks | |
| 14:53:08 | kashyap | sean-k-mooney: Yes, the version is chosen in a way that the operator must ensure the Compute nodes are capable of it | |
| 14:53:40 | kashyap | sean-k-mooney: Recall, the version in the machien type matches CentOS|RHEL minor version - it is pretty clear to figure out the problem due to host OS version mismatch... | |
| 14:53:46 | sean-k-mooney | yep wich i ugess is the max verion supported by the oldes hypervior | |
| 14:54:02 | sean-k-mooney | which would be the verion form 8.0 in this case | |
| 14:54:41 | kashyap | Right. | |
| 14:54:46 | kashyap | We clear? :) | |
| 14:57:40 | kashyap | I take it we are. | |
| 15:02:51 | sean-k-mooney | kashyap: yep sorry was talking dowstream | |
| 15:03:09 | sean-k-mooney | and ya we are on the same page regarding the requirements here | |
| 15:03:31 | kashyap | No problem; thanks | |
| 15:10:21 | lyarwood | kashyap / sean-k-mooney ; silly question but why wouldn't we use the q35 alias all the time downstream? | |
| 15:10:42 | kashyap | lyarwood: Live migration compatibility | |
| 15:10:51 | lyarwood | kashyap: but LM wouldn't switch between the two surely? | |
| 15:10:53 | kashyap | lyarwood: I have an example written down, with a scenario, so you can see it plainly | |
| 15:11:04 | kashyap | Let me post it somewhere, so you can see it, and possibly punch holes in it | |
| 15:12:06 | lyarwood | ah wait so QEMU would just blindly use the alias on the dest as well? | |
| 15:12:13 | lyarwood | even if it pointed to a newer version | |
| 15:12:43 | lyarwood | sorry I assumed libvirt/QEMU would translate the alias into an actual versioned q35 machine type and stick with that for the life cycle of the domain | |
| 15:14:58 | kashyap | lyarwood: Yes | |
| 15:15:32 | kashyap | lyarwood: If you try to migrate a guest with RHEL-7.7 machine type (running on RHEL-7 COmpute node) to a RHEL-7.6 Compute node -- it fails spectacularly | |
| 15:15:45 | kashyap | Because 7.6 host obviously doesn't know about 7.7 :) | |
| 15:16:47 | lyarwood | right but my point is more that forward migration from 7.6 to 7.7 just using the q35 alias could also fail | |
| 15:17:03 | lyarwood | as the domain would jump from the 7.6 to 7.7 version of the machine type | |
| 15:17:05 | kashyap | lyarwood: https://kashyapc.fedorapeople.org/versioned-machine-types-and-live-migration-gotcha.txt | |
| 15:17:08 | lyarwood | well maybe not fail but things could change | |
| 15:17:27 | kashyap | lyarwood: Forward migration fomr 7.6 to 7.7 will _not_ fail | |
| 15:17:28 | kashyap | Why? | |
| 15:17:55 | kashyap | Because, 7.7 host will have all the older (7.6, 7.5, 7.4) machine types precisely for live migration compat | |
| 15:18:07 | kashyap | lyarwood: I had the above text somewhere in the spec, but deleted it, for "brevity" | |
| 15:18:23 | kashyap | artom: Since you also spent non-trivial time reviewing the spec, you might benefit from reading this: https://kashyapc.fedorapeople.org/versioned-machine-types-and-live-migration-gotcha.txt | |
| 15:18:42 | lyarwood | It will now be _impossible_ to migrate a guest from a RHEL 7.6-based | |
| 15:18:43 | lyarwood | compute node to a RHEL 7.7-based — because RHEL 7.6 won't know about | |
| 15:18:45 | lyarwood | the RHEL 7.7's machine type. | |
| 15:18:56 | lyarwood | kashyap: ^ that was my point sorry | |
| 15:19:25 | lyarwood | and the workaround appears to be to pin to the oldest version | |
| 15:19:33 | kashyap | Yep | |
| 15:19:40 | lyarwood | but I just don't get how this works downstream across multple releases | |
| 15:19:49 | lyarwood | we'd basically need to pin to the first version used in the env | |
| 15:20:55 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Fix list rendering in the accelerator support doc https://review.opendev.org/721846 | |
| 15:21:58 | lyarwood | surely we should work out and persist the versioned machine type during a migration? | |
| 15:22:24 | lyarwood | so if the source is 8.1 and using the q35 alias ensure it uses that on the dest | |
| 15:22:36 | lyarwood | but all new instances use the 8.2 q35 type? | |
| 15:22:56 | kashyap | lyarwood: For downstream, yes: you need to pick a versioned machine type and set it across the nodes before deplyoying. | |
| 15:22:59 | aarents | lyarwood: kashyap we hit this kind of issues.. but Im not the guy who work on it I will check how we dealt with that.. | |
| 15:23:01 | lyarwood | kashyap: sorry I'm talking out loud here, could you add the LM section back into the spec and I'll add words there. | |
| 15:23:26 | kashyap | lyarwood: Sure, let me add that bit in the spec in a separate section; it's important | |
| 15:23:30 | lyarwood | kashyap: yeah the issue with that is you can never move from it | |
| 15:24:16 | kashyap | lyarwood: Hang on - you _can_ move; it's a step-by-step process: | |
| 15:25:35 | kashyap | lyarwood: Err, my "Here's the problem" has a bug! | |
| 15:25:41 | kashyap | It should read the _other_ way round. | |
| 15:25:43 | kashyap | Let me update it | |
| 15:25:59 | lyarwood | kashyap: kk, lets get the spec updated and chat there :) | |
| 15:26:02 | kashyap | You cannot migrte from 7.7 to 7.6 | |
| 15:26:14 | kashyap | But you can jolly well migrate from 7.6 to 7.7 :) | |
| 15:26:15 | kashyap | Yes | |
| 15:26:17 | kashyap | A sec | |
| 15:30:33 | artom | I feel like this whole mess could be solved if we tracked machine type on a per-instance basis | |
| 15:30:33 | kashyap | aarents: Noted :) | |
| 15:31:00 | artom | Though the problem that raises is live-migration from an N+1 to an N compute | |
| 15:31:14 | artom | The dest N might now know the machine type, as it's newer | |
| 15:31:37 | kashyap | Per-instance tracking is just too damn fragile and messy | |
| 15:31:47 | kashyap | I'd say per-Compute based is better for debugging, too. | |
| 15:31:51 | artom | Maybe, but how/why? | |
| 15:31:55 | kashyap | We should not forget the Operator's predicmanet | |
| 15:32:02 | kashyap | s/predicmanet/predicament/ | |
| 15:32:10 | artom | This whole discussion shows that per-compute isn't super clean either :) | |
| 15:32:22 | kashyap | artom: The live migration from N+1 to N always remains a problem. | |
| 15:32:54 | artom | True | |
| 15:33:06 | artom | Could we not incorporate machine type into our LM checks? | |
| 15:33:10 | kashyap | Can you tell more what do we gain more per-instance tracking solve? Just asking sincerely... :) | |