| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-23 | |||
| 14:30:14 | dansmith | un-WIP and fix? | |
| 14:30:20 | artom | Yeah, fix was implied :) | |
| 14:30:28 | dansmith | juuuuust checking :) | |
| 14:30:43 | artom | Haha, there are limits to my lazyness ;) | |
| 14:30:57 | dansmith | well, you can see my level of confidence in your laziness I guess | |
| 14:31:50 | artom | ;_; | |
| 14:32:10 | dansmith | efried: are you a candidate reviewer for this numa live migration stuff? seems like it's probably somewhat in your wheelhouse and sphere of interest | |
| 14:32:36 | dougsz | In cells v2, do you run online_data_migrations only when *all* cells are upgraded? | |
| 14:32:55 | dansmith | dougsz: you can run them against individual cells as they're upgraded or afterwards against all of them | |
| 14:33:40 | AJaeger | efried, so are fine with https://review.opendev.org/677178, https://review.opendev.org/677176 and https://review.opendev.org/677177, please ? | |
| 14:34:25 | efried | dansmith: sphere of interest, as dictated by my employer, yes. Wheelhouse, not in the slightest. I've been hoping people who actually know wtf they're looking at will do the dirty work here. | |
| 14:35:02 | efried | AJaeger: I'm not stable. dansmith might be able to push those for you, though. | |
| 14:35:03 | dansmith | efried: um, it's numa and little bit of rpc | |
| 14:35:28 | dansmith | efried: but I'm trying to make sure it's clean and then will have to look for another reviewer to +W and figured that might be you | |
| 14:35:46 | dansmith | efried: it's really not very complicated.. much less so than I originally thought when discussing the meta details | |
| 14:35:48 | dougsz | dansmith: thanks - so to upgrade one cell, i run online_data_migrations in the cell, and then again at the top level? | |
| 14:36:29 | dansmith | dougsz: "run again at the top level" meaning "run again against all cells once they're all updated" ? | |
| 14:36:46 | efried | dansmith: I will dig in if I need to, but I'm still way low on the learning curve of both numa and rpc. | |
| 14:36:51 | mriedem | alex_xu: ack, and replied | |
| 14:36:55 | dansmith | efried: opportunity? :) | |
| 14:37:00 | mriedem | alex_xu: tl;dr do whatever you want | |
| 14:37:00 | efried | yeah | |
| 14:37:36 | dansmith | heh | |
| 14:38:42 | alex_xu | I want to sleep now :) | |
| 14:39:12 | artom | efried, FWIW, the crux of it is converting parts of https://libvirt.org/formatdomain.html#elementsNUMATuning and https://libvirt.org/formatdomain.html#elementsCPUTuning to objects and back | |
| 14:39:46 | artom | So it's not touching any of the nasty code in nova/virt/hardware.py | |
| 14:40:37 | artom | Or just get alex_xu to do it, if he's more up to speed on the libvirt'y things ;) | |
| 14:40:46 | mriedem | dansmith: i'm likely your other core for numa live migration | |
| 14:40:58 | AJaeger | efried: ah, thanks | |
| 14:41:04 | dansmith | mriedem: I didn't think you'd want to | |
| 14:41:14 | dansmith | mriedem: and/or I didn't want to know what pound of flesh you'd want in return :D | |
| 14:41:15 | mriedem | i've gone through it before...in rocky probably at tihs point | |
| 14:41:27 | dansmith | yeah, I know | |
| 14:41:46 | mriedem | and i'm more familiar with the live migration and rt claims flows that many | |
| 14:41:51 | mriedem | *than | |
| 14:41:59 | dansmith | I know, but .. see above :) | |
| 14:42:07 | mriedem | however, i also have confidence issues with it | |
| 14:42:46 | mriedem | so i've been waiting for another core to dig in first, root out the obvious stuff, and someone (intel/windriver) to come along and say they've actually tested it in a lab | |
| 14:42:57 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Mask the token used to allow access to consoles https://review.opendev.org/220622 | |
| 14:43:01 | dansmith | well, I'm doing the former | |
| 14:43:04 | mriedem | yeah, | |
| 14:43:20 | dansmith | and I think it's just going to be artom's hand testing | |
| 14:43:21 | mriedem | and i think as of last friday sean-k-mooney said in his free time at home when he shouldn't be on his own personal hardware was trying to do the latter | |
| 14:43:26 | dansmith | at least initially | |
| 14:43:36 | dansmith | or that | |
| 14:43:43 | artom | mriedem, Dean told me he's brought it up the chain to try and get resources to test it, but not sure that's going to pan out | |
| 14:43:57 | mriedem | i know intel is strapped for cash and resources these days | |
| 14:43:58 | mriedem | it's sad | |
| 14:44:02 | mriedem | sad folks | |
| 14:44:03 | alex_xu | mriedem: thanks anyway, sorry for pushing hard | |
| 14:44:06 | sean-k-mooney | reads back | |
| 14:44:11 | dansmith | it's not going to happen in time to land this either I'm sure | |
| 14:44:12 | dansmith | lol | |
| 14:44:16 | dougsz | dansmith: maybe I'm confused. If I run `nova-manage db online_data_migrations` using the cell conductor nova.conf, I update only the cell, if I run `nova-manage db online_data_migrations` using the top level nova.conf, I upgrade all cells? | |
| 14:44:38 | dansmith | dougsz: no, hence my question ;) | |
| 14:44:40 | artom | FWIW, I do have that whitebox tempest plugin code | |
| 14:44:50 | artom | It's nowhere upstream yet, but that's next on my list | |
| 14:44:57 | dansmith | dougsz: the former per cell is how it works, which is why I asked why you'd run again "at the top level" | |
| 14:45:04 | alex_xu | artom: emm..not sure what is it yet | |
| 14:45:28 | artom | Move whitebox under openstack-qa, find hardware on which to run it. Fort Nebula is a good candidate | |
| 14:45:36 | sean-k-mooney | i do plan to test artoms stuff next week. i need to resintall the os on those servers. but i have already set up one of them | |
| 14:45:56 | AJaeger | dansmith: could you put the following three on your review list, please? https://review.opendev.org/677178, https://review.opendev.org/677176 and https://review.opendev.org/677177 all remove an experimental opensuse job for an eol distro from stable branches | |
| 14:45:58 | mriedem | dougsz: iow, you have to run that command per cell | |
| 14:46:25 | dansmith | mriedem: he said he used the cell conductor config, which is the per-cell bit, so I think he gets that | |
| 14:46:50 | mriedem | ok | |
| 14:46:54 | dansmith | I guess we do have some migrations in the api database, | |
| 14:47:15 | dansmith | so I guess running against the api config is also necessary, I dunno why I wasn't thinking about that initially | |
| 14:47:17 | dansmith | dougsz: ^ | |
| 14:48:23 | artom | alex_xu, NUMA live migration, you looked at it a bit before, and there's some overlap with your vpmem work I think. dansmith and I are hunting for a second non-RH core | |
| 14:48:25 | dougsz | dansmith, mriedem: thanks, I think that makes sense. So per cell, *and* at the top level to do it everywhere. | |
| 14:48:41 | mriedem | "to do it everywhere" isn't a thing | |
| 14:48:48 | mriedem | the command doesn't broadcast across all cells | |
| 14:48:52 | dansmith | right, | |
| 14:48:57 | dansmith | that's the part I was taking exception with | |
| 14:49:01 | mriedem | dan is just saying there are some migrations in the API DB which is "at the top" | |
| 14:49:08 | mriedem | "top of the topology" iow | |
| 14:49:10 | alex_xu | artom: yea, i can onboard the review, probably need one or two day, rampup your patches | |
| 14:49:30 | artom | alex_xu, thank you, appreciated :) | |
| 14:50:17 | dougsz | mriedem: Ok, thanks, and I trigger those top level migrations via nova-manage api_db sync. I think I get it | |
| 14:50:21 | alex_xu | artom: btw, I don't think it will overlap with vpmem anymore, but will check again | |
| 14:50:22 | openstackgerrit | Eric Fried proposed openstack/nova master: Update SDK fixture for openstacksdk 0.35.0 https://review.opendev.org/678237 | |
| 14:50:31 | efried | mriedem: more sdk turducken :( ^ | |
| 14:50:54 | mriedem | dougsz: online_data_migrations is not the same as the sync command | |
| 14:51:03 | mriedem | sync == upgrade db table schemas | |
| 14:51:24 | mriedem | online_data_migrations == migrate existing data in the table to some new format if necessary | |
| 14:51:31 | dougsz | got it | |
| 14:53:57 | dougsz | final question, If I upgrade a single cell, to get it to move to latest RPC version, I need to SIG-HUP the top level Nova services as well? | |
| 14:54:40 | mriedem | AJaeger: i got them | |
| 14:54:52 | efried | SIGHUP is still broken, until https://review.opendev.org/#/c/641907/ | |
| 14:55:18 | AJaeger | thanks, mriedem ! | |
| 14:55:39 | dougsz | yeah, just thinking, I upgrade one cell, I need to restart all top level services because SIG-HUP doesn't work | |
| 14:55:46 | mriedem | dougsz: well, you should be pinning the rpc upgrade level for the computes at the top level so when you upgrade your computes in the cell, the top level (api/conductor) isn't sending messages the computes won't understand | |
| 14:56:01 | mriedem | dougsz: are you updating the control plane services first or last? | |
| 14:56:12 | dougsz | control plane first | |
| 14:56:15 | efried | stephenfin: I'm waiting for a new push of the cpu-resources series, right? | |
| 14:56:25 | mriedem | https://docs.openstack.org/nova/latest/user/upgrade.html | |
| 14:56:26 | stephenfin | yup | |
| 14:56:29 | mriedem | dougsz: https://docs.openstack.org/nova/latest/user/upgrade.html | |
| 14:56:41 | stephenfin | working on it as we speak | |
| 14:56:41 | dougsz | yeah, trying to follow that in a cells context | |