| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-23 | |||
| 14:43:04 | mriedem | yeah, | |
| 14:43:20 | dansmith | and I think it's just going to be artom's hand testing | |
| 14:43:21 | mriedem | and i think as of last friday sean-k-mooney said in his free time at home when he shouldn't be on his own personal hardware was trying to do the latter | |
| 14:43:26 | dansmith | at least initially | |
| 14:43:36 | dansmith | or that | |
| 14:43:43 | artom | mriedem, Dean told me he's brought it up the chain to try and get resources to test it, but not sure that's going to pan out | |
| 14:43:57 | mriedem | i know intel is strapped for cash and resources these days | |
| 14:43:58 | mriedem | it's sad | |
| 14:44:02 | mriedem | sad folks | |
| 14:44:03 | alex_xu | mriedem: thanks anyway, sorry for pushing hard | |
| 14:44:06 | sean-k-mooney | reads back | |
| 14:44:11 | dansmith | it's not going to happen in time to land this either I'm sure | |
| 14:44:12 | dansmith | lol | |
| 14:44:16 | dougsz | dansmith: maybe I'm confused. If I run `nova-manage db online_data_migrations` using the cell conductor nova.conf, I update only the cell, if I run `nova-manage db online_data_migrations` using the top level nova.conf, I upgrade all cells? | |
| 14:44:38 | dansmith | dougsz: no, hence my question ;) | |
| 14:44:40 | artom | FWIW, I do have that whitebox tempest plugin code | |
| 14:44:50 | artom | It's nowhere upstream yet, but that's next on my list | |
| 14:44:57 | dansmith | dougsz: the former per cell is how it works, which is why I asked why you'd run again "at the top level" | |
| 14:45:04 | alex_xu | artom: emm..not sure what is it yet | |
| 14:45:28 | artom | Move whitebox under openstack-qa, find hardware on which to run it. Fort Nebula is a good candidate | |
| 14:45:36 | sean-k-mooney | i do plan to test artoms stuff next week. i need to resintall the os on those servers. but i have already set up one of them | |
| 14:45:56 | AJaeger | dansmith: could you put the following three on your review list, please? https://review.opendev.org/677178, https://review.opendev.org/677176 and https://review.opendev.org/677177 all remove an experimental opensuse job for an eol distro from stable branches | |
| 14:45:58 | mriedem | dougsz: iow, you have to run that command per cell | |
| 14:46:25 | dansmith | mriedem: he said he used the cell conductor config, which is the per-cell bit, so I think he gets that | |
| 14:46:50 | mriedem | ok | |
| 14:46:54 | dansmith | I guess we do have some migrations in the api database, | |
| 14:47:15 | dansmith | so I guess running against the api config is also necessary, I dunno why I wasn't thinking about that initially | |
| 14:47:17 | dansmith | dougsz: ^ | |
| 14:48:23 | artom | alex_xu, NUMA live migration, you looked at it a bit before, and there's some overlap with your vpmem work I think. dansmith and I are hunting for a second non-RH core | |
| 14:48:25 | dougsz | dansmith, mriedem: thanks, I think that makes sense. So per cell, *and* at the top level to do it everywhere. | |
| 14:48:41 | mriedem | "to do it everywhere" isn't a thing | |
| 14:48:48 | mriedem | the command doesn't broadcast across all cells | |
| 14:48:52 | dansmith | right, | |
| 14:48:57 | dansmith | that's the part I was taking exception with | |
| 14:49:01 | mriedem | dan is just saying there are some migrations in the API DB which is "at the top" | |
| 14:49:08 | mriedem | "top of the topology" iow | |
| 14:49:10 | alex_xu | artom: yea, i can onboard the review, probably need one or two day, rampup your patches | |
| 14:49:30 | artom | alex_xu, thank you, appreciated :) | |
| 14:50:17 | dougsz | mriedem: Ok, thanks, and I trigger those top level migrations via nova-manage api_db sync. I think I get it | |
| 14:50:21 | alex_xu | artom: btw, I don't think it will overlap with vpmem anymore, but will check again | |
| 14:50:22 | openstackgerrit | Eric Fried proposed openstack/nova master: Update SDK fixture for openstacksdk 0.35.0 https://review.opendev.org/678237 | |
| 14:50:31 | efried | mriedem: more sdk turducken :( ^ | |
| 14:50:54 | mriedem | dougsz: online_data_migrations is not the same as the sync command | |
| 14:51:03 | mriedem | sync == upgrade db table schemas | |
| 14:51:24 | mriedem | online_data_migrations == migrate existing data in the table to some new format if necessary | |
| 14:51:31 | dougsz | got it | |
| 14:53:57 | dougsz | final question, If I upgrade a single cell, to get it to move to latest RPC version, I need to SIG-HUP the top level Nova services as well? | |
| 14:54:40 | mriedem | AJaeger: i got them | |
| 14:54:52 | efried | SIGHUP is still broken, until https://review.opendev.org/#/c/641907/ | |
| 14:55:18 | AJaeger | thanks, mriedem ! | |
| 14:55:39 | dougsz | yeah, just thinking, I upgrade one cell, I need to restart all top level services because SIG-HUP doesn't work | |
| 14:55:46 | mriedem | dougsz: well, you should be pinning the rpc upgrade level for the computes at the top level so when you upgrade your computes in the cell, the top level (api/conductor) isn't sending messages the computes won't understand | |
| 14:56:01 | mriedem | dougsz: are you updating the control plane services first or last? | |
| 14:56:12 | dougsz | control plane first | |
| 14:56:15 | efried | stephenfin: I'm waiting for a new push of the cpu-resources series, right? | |
| 14:56:25 | mriedem | https://docs.openstack.org/nova/latest/user/upgrade.html | |
| 14:56:26 | stephenfin | yup | |
| 14:56:29 | mriedem | dougsz: https://docs.openstack.org/nova/latest/user/upgrade.html | |
| 14:56:41 | stephenfin | working on it as we speak | |
| 14:56:41 | dougsz | yeah, trying to follow that in a cells context | |
| 14:56:52 | efried | stephenfin: I'm hoping that if I read the code once a day for a week, I'll understand it. | |
| 14:57:16 | efried | Currently it's you making changes I don't understand to code I don't understand. | |
| 14:57:42 | mriedem | dougsz: cellsv2 yeah? | |
| 14:57:45 | dougsz | yeah | |
| 14:57:50 | mriedem | dougsz: what services are running in the cells? just nova-compute? | |
| 14:58:12 | dougsz | + nova-ssh, nova-conductor | |
| 14:58:23 | mriedem | idk what nova-ssh is | |
| 14:59:07 | mriedem | anyway, if you upgrade the control plane first, you want to make sure those are configured for [upgrade_levels]/compute (either auto or whatever your n-1 release you're upgrading from is) | |
| 14:59:20 | mriedem | that way the controller services won't send N version messages to N-1 services | |
| 14:59:42 | mriedem | i.e. pin the rpc versions to the lowest common version, roll through upgrades, and then unpin | |
| 15:00:42 | dougsz | ok, thanks mriedem, that bit makes sense. I will go and have a play | |
| 15:12:11 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: allow getting resource request of every bound ports of an instance https://review.opendev.org/655110 | |
| 15:14:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass network API to the conducor's MigrationTask https://review.opendev.org/655111 | |
| 15:16:45 | dansmith | efried: sighup still does the thing he needs, just inefficiently, IIRC | |
| 15:16:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add request_spec to server move RPC calls https://review.opendev.org/655721 | |
| 15:16:58 | dansmith | efried: it effectively restarts the whole process, which is good enough, just too heavy | |
| 15:17:15 | efried | dansmith: I thought we decided SIGHUP also breaks privsep | |
| 15:17:44 | dansmith | oh does it? | |
| 15:18:08 | efried | something about either restarting or not restarting the privsep daemon... | |
| 15:18:12 | dougsz | https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/handlers/main.yml#L211 | |
| 15:18:12 | efried | now I can't remember where I saw it. | |
| 15:18:25 | dougsz | dansmith efried ^ That's the issue afaik | |
| 15:18:26 | dansmith | not restarting is fine, as long as it doesn't lose a reference to the pipe, which I thought was global | |
| 15:19:10 | dansmith | dougsz: oh, right, right, ignore me | |
| 15:19:16 | dansmith | I forgot the rabbit hole went deeper | |
| 15:19:17 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: re-calculate provider mapping during migration https://review.opendev.org/655112 | |
| 15:19:22 | dougsz | np! | |
| 15:19:34 | dansmith | the simple case is that it does a full restart for no reason, but I forgot it also breaks some state as a result | |
| 15:20:41 | efried | someone stephenfin with oslo-core stephenfin should merge that patch | |
| 15:21:19 | stephenfin | whuh? | |
| 15:21:31 | efried | https://review.opendev.org/#/c/641907/ | |
| 15:21:40 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: update allocation in binding profile during migrate https://review.opendev.org/656422 | |
| 15:22:18 | stephenfin | I'm going to finish this cpu-resources reshuffle, but I can take a look after that | |
| 15:22:33 | efried | cool. No hurry, clearly. It's been open since like rocky. | |
| 15:22:49 | efried | sure would be nice to stop having to tell people SIGHUP is broken tho. | |
| 15:22:50 | beekneemech | If you figure it out you'll be ahead of the rest of us. :-) | |
| 15:23:19 | efried | (oh, did I trigger a keyword?) | |
| 15:23:26 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: doc: remove confusing docs about aggregate allocation ratios https://review.opendev.org/678254 | |
| 15:23:48 | beekneemech | Yeah, I added an oslo highlight after the Denver PTL discussion. | |
| 15:24:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extend NeutronFixture to handle migrations https://review.opendev.org/655114 | |