| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-23 | |||
| 14:35:28 | dansmith | efried: but I'm trying to make sure it's clean and then will have to look for another reviewer to +W and figured that might be you | |
| 14:35:46 | dansmith | efried: it's really not very complicated.. much less so than I originally thought when discussing the meta details | |
| 14:35:48 | dougsz | dansmith: thanks - so to upgrade one cell, i run online_data_migrations in the cell, and then again at the top level? | |
| 14:36:29 | dansmith | dougsz: "run again at the top level" meaning "run again against all cells once they're all updated" ? | |
| 14:36:46 | efried | dansmith: I will dig in if I need to, but I'm still way low on the learning curve of both numa and rpc. | |
| 14:36:51 | mriedem | alex_xu: ack, and replied | |
| 14:36:55 | dansmith | efried: opportunity? :) | |
| 14:37:00 | mriedem | alex_xu: tl;dr do whatever you want | |
| 14:37:00 | efried | yeah | |
| 14:37:36 | dansmith | heh | |
| 14:38:42 | alex_xu | I want to sleep now :) | |
| 14:39:12 | artom | efried, FWIW, the crux of it is converting parts of https://libvirt.org/formatdomain.html#elementsNUMATuning and https://libvirt.org/formatdomain.html#elementsCPUTuning to objects and back | |
| 14:39:46 | artom | So it's not touching any of the nasty code in nova/virt/hardware.py | |
| 14:40:37 | artom | Or just get alex_xu to do it, if he's more up to speed on the libvirt'y things ;) | |
| 14:40:46 | mriedem | dansmith: i'm likely your other core for numa live migration | |
| 14:40:58 | AJaeger | efried: ah, thanks | |
| 14:41:04 | dansmith | mriedem: I didn't think you'd want to | |
| 14:41:14 | dansmith | mriedem: and/or I didn't want to know what pound of flesh you'd want in return :D | |
| 14:41:15 | mriedem | i've gone through it before...in rocky probably at tihs point | |
| 14:41:27 | dansmith | yeah, I know | |
| 14:41:46 | mriedem | and i'm more familiar with the live migration and rt claims flows that many | |
| 14:41:51 | mriedem | *than | |
| 14:41:59 | dansmith | I know, but .. see above :) | |
| 14:42:07 | mriedem | however, i also have confidence issues with it | |
| 14:42:46 | mriedem | so i've been waiting for another core to dig in first, root out the obvious stuff, and someone (intel/windriver) to come along and say they've actually tested it in a lab | |
| 14:42:57 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Mask the token used to allow access to consoles https://review.opendev.org/220622 | |
| 14:43:01 | dansmith | well, I'm doing the former | |
| 14:43:04 | mriedem | yeah, | |
| 14:43:20 | dansmith | and I think it's just going to be artom's hand testing | |
| 14:43:21 | mriedem | and i think as of last friday sean-k-mooney said in his free time at home when he shouldn't be on his own personal hardware was trying to do the latter | |
| 14:43:26 | dansmith | at least initially | |
| 14:43:36 | dansmith | or that | |
| 14:43:43 | artom | mriedem, Dean told me he's brought it up the chain to try and get resources to test it, but not sure that's going to pan out | |
| 14:43:57 | mriedem | i know intel is strapped for cash and resources these days | |
| 14:43:58 | mriedem | it's sad | |
| 14:44:02 | mriedem | sad folks | |
| 14:44:03 | alex_xu | mriedem: thanks anyway, sorry for pushing hard | |
| 14:44:06 | sean-k-mooney | reads back | |
| 14:44:11 | dansmith | it's not going to happen in time to land this either I'm sure | |
| 14:44:12 | dansmith | lol | |
| 14:44:16 | dougsz | dansmith: maybe I'm confused. If I run `nova-manage db online_data_migrations` using the cell conductor nova.conf, I update only the cell, if I run `nova-manage db online_data_migrations` using the top level nova.conf, I upgrade all cells? | |
| 14:44:38 | dansmith | dougsz: no, hence my question ;) | |
| 14:44:40 | artom | FWIW, I do have that whitebox tempest plugin code | |
| 14:44:50 | artom | It's nowhere upstream yet, but that's next on my list | |
| 14:44:57 | dansmith | dougsz: the former per cell is how it works, which is why I asked why you'd run again "at the top level" | |
| 14:45:04 | alex_xu | artom: emm..not sure what is it yet | |
| 14:45:28 | artom | Move whitebox under openstack-qa, find hardware on which to run it. Fort Nebula is a good candidate | |
| 14:45:36 | sean-k-mooney | i do plan to test artoms stuff next week. i need to resintall the os on those servers. but i have already set up one of them | |
| 14:45:56 | AJaeger | dansmith: could you put the following three on your review list, please? https://review.opendev.org/677178, https://review.opendev.org/677176 and https://review.opendev.org/677177 all remove an experimental opensuse job for an eol distro from stable branches | |
| 14:45:58 | mriedem | dougsz: iow, you have to run that command per cell | |
| 14:46:25 | dansmith | mriedem: he said he used the cell conductor config, which is the per-cell bit, so I think he gets that | |
| 14:46:50 | mriedem | ok | |
| 14:46:54 | dansmith | I guess we do have some migrations in the api database, | |
| 14:47:15 | dansmith | so I guess running against the api config is also necessary, I dunno why I wasn't thinking about that initially | |
| 14:47:17 | dansmith | dougsz: ^ | |
| 14:48:23 | artom | alex_xu, NUMA live migration, you looked at it a bit before, and there's some overlap with your vpmem work I think. dansmith and I are hunting for a second non-RH core | |
| 14:48:25 | dougsz | dansmith, mriedem: thanks, I think that makes sense. So per cell, *and* at the top level to do it everywhere. | |
| 14:48:41 | mriedem | "to do it everywhere" isn't a thing | |
| 14:48:48 | mriedem | the command doesn't broadcast across all cells | |
| 14:48:52 | dansmith | right, | |
| 14:48:57 | dansmith | that's the part I was taking exception with | |
| 14:49:01 | mriedem | dan is just saying there are some migrations in the API DB which is "at the top" | |
| 14:49:08 | mriedem | "top of the topology" iow | |
| 14:49:10 | alex_xu | artom: yea, i can onboard the review, probably need one or two day, rampup your patches | |
| 14:49:30 | artom | alex_xu, thank you, appreciated :) | |
| 14:50:17 | dougsz | mriedem: Ok, thanks, and I trigger those top level migrations via nova-manage api_db sync. I think I get it | |
| 14:50:21 | alex_xu | artom: btw, I don't think it will overlap with vpmem anymore, but will check again | |
| 14:50:22 | openstackgerrit | Eric Fried proposed openstack/nova master: Update SDK fixture for openstacksdk 0.35.0 https://review.opendev.org/678237 | |
| 14:50:31 | efried | mriedem: more sdk turducken :( ^ | |
| 14:50:54 | mriedem | dougsz: online_data_migrations is not the same as the sync command | |
| 14:51:03 | mriedem | sync == upgrade db table schemas | |
| 14:51:24 | mriedem | online_data_migrations == migrate existing data in the table to some new format if necessary | |
| 14:51:31 | dougsz | got it | |
| 14:53:57 | dougsz | final question, If I upgrade a single cell, to get it to move to latest RPC version, I need to SIG-HUP the top level Nova services as well? | |
| 14:54:40 | mriedem | AJaeger: i got them | |
| 14:54:52 | efried | SIGHUP is still broken, until https://review.opendev.org/#/c/641907/ | |
| 14:55:18 | AJaeger | thanks, mriedem ! | |
| 14:55:39 | dougsz | yeah, just thinking, I upgrade one cell, I need to restart all top level services because SIG-HUP doesn't work | |
| 14:55:46 | mriedem | dougsz: well, you should be pinning the rpc upgrade level for the computes at the top level so when you upgrade your computes in the cell, the top level (api/conductor) isn't sending messages the computes won't understand | |
| 14:56:01 | mriedem | dougsz: are you updating the control plane services first or last? | |
| 14:56:12 | dougsz | control plane first | |
| 14:56:15 | efried | stephenfin: I'm waiting for a new push of the cpu-resources series, right? | |
| 14:56:25 | mriedem | https://docs.openstack.org/nova/latest/user/upgrade.html | |
| 14:56:26 | stephenfin | yup | |
| 14:56:29 | mriedem | dougsz: https://docs.openstack.org/nova/latest/user/upgrade.html | |
| 14:56:41 | stephenfin | working on it as we speak | |
| 14:56:41 | dougsz | yeah, trying to follow that in a cells context | |
| 14:56:52 | efried | stephenfin: I'm hoping that if I read the code once a day for a week, I'll understand it. | |
| 14:57:16 | efried | Currently it's you making changes I don't understand to code I don't understand. | |
| 14:57:42 | mriedem | dougsz: cellsv2 yeah? | |
| 14:57:45 | dougsz | yeah | |
| 14:57:50 | mriedem | dougsz: what services are running in the cells? just nova-compute? | |
| 14:58:12 | dougsz | + nova-ssh, nova-conductor | |
| 14:58:23 | mriedem | idk what nova-ssh is | |
| 14:59:07 | mriedem | anyway, if you upgrade the control plane first, you want to make sure those are configured for [upgrade_levels]/compute (either auto or whatever your n-1 release you're upgrading from is) | |
| 14:59:20 | mriedem | that way the controller services won't send N version messages to N-1 services | |
| 14:59:42 | mriedem | i.e. pin the rpc versions to the lowest common version, roll through upgrades, and then unpin | |
| 15:00:42 | dougsz | ok, thanks mriedem, that bit makes sense. I will go and have a play | |
| 15:12:11 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: allow getting resource request of every bound ports of an instance https://review.opendev.org/655110 | |
| 15:14:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass network API to the conducor's MigrationTask https://review.opendev.org/655111 | |