| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-23 | |||
| 14:49:30 | artom | alex_xu, thank you, appreciated :) | |
| 14:50:17 | dougsz | mriedem: Ok, thanks, and I trigger those top level migrations via nova-manage api_db sync. I think I get it | |
| 14:50:21 | alex_xu | artom: btw, I don't think it will overlap with vpmem anymore, but will check again | |
| 14:50:22 | openstackgerrit | Eric Fried proposed openstack/nova master: Update SDK fixture for openstacksdk 0.35.0 https://review.opendev.org/678237 | |
| 14:50:31 | efried | mriedem: more sdk turducken :( ^ | |
| 14:50:54 | mriedem | dougsz: online_data_migrations is not the same as the sync command | |
| 14:51:03 | mriedem | sync == upgrade db table schemas | |
| 14:51:24 | mriedem | online_data_migrations == migrate existing data in the table to some new format if necessary | |
| 14:51:31 | dougsz | got it | |
| 14:53:57 | dougsz | final question, If I upgrade a single cell, to get it to move to latest RPC version, I need to SIG-HUP the top level Nova services as well? | |
| 14:54:40 | mriedem | AJaeger: i got them | |
| 14:54:52 | efried | SIGHUP is still broken, until https://review.opendev.org/#/c/641907/ | |
| 14:55:18 | AJaeger | thanks, mriedem ! | |
| 14:55:39 | dougsz | yeah, just thinking, I upgrade one cell, I need to restart all top level services because SIG-HUP doesn't work | |
| 14:55:46 | mriedem | dougsz: well, you should be pinning the rpc upgrade level for the computes at the top level so when you upgrade your computes in the cell, the top level (api/conductor) isn't sending messages the computes won't understand | |
| 14:56:01 | mriedem | dougsz: are you updating the control plane services first or last? | |
| 14:56:12 | dougsz | control plane first | |
| 14:56:15 | efried | stephenfin: I'm waiting for a new push of the cpu-resources series, right? | |
| 14:56:25 | mriedem | https://docs.openstack.org/nova/latest/user/upgrade.html | |
| 14:56:26 | stephenfin | yup | |
| 14:56:29 | mriedem | dougsz: https://docs.openstack.org/nova/latest/user/upgrade.html | |
| 14:56:41 | stephenfin | working on it as we speak | |
| 14:56:41 | dougsz | yeah, trying to follow that in a cells context | |
| 14:56:52 | efried | stephenfin: I'm hoping that if I read the code once a day for a week, I'll understand it. | |
| 14:57:16 | efried | Currently it's you making changes I don't understand to code I don't understand. | |
| 14:57:42 | mriedem | dougsz: cellsv2 yeah? | |
| 14:57:45 | dougsz | yeah | |
| 14:57:50 | mriedem | dougsz: what services are running in the cells? just nova-compute? | |
| 14:58:12 | dougsz | + nova-ssh, nova-conductor | |
| 14:58:23 | mriedem | idk what nova-ssh is | |
| 14:59:07 | mriedem | anyway, if you upgrade the control plane first, you want to make sure those are configured for [upgrade_levels]/compute (either auto or whatever your n-1 release you're upgrading from is) | |
| 14:59:20 | mriedem | that way the controller services won't send N version messages to N-1 services | |
| 14:59:42 | mriedem | i.e. pin the rpc versions to the lowest common version, roll through upgrades, and then unpin | |
| 15:00:42 | dougsz | ok, thanks mriedem, that bit makes sense. I will go and have a play | |
| 15:12:11 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: allow getting resource request of every bound ports of an instance https://review.opendev.org/655110 | |
| 15:14:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass network API to the conducor's MigrationTask https://review.opendev.org/655111 | |
| 15:16:45 | dansmith | efried: sighup still does the thing he needs, just inefficiently, IIRC | |
| 15:16:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add request_spec to server move RPC calls https://review.opendev.org/655721 | |
| 15:16:58 | dansmith | efried: it effectively restarts the whole process, which is good enough, just too heavy | |
| 15:17:15 | efried | dansmith: I thought we decided SIGHUP also breaks privsep | |
| 15:17:44 | dansmith | oh does it? | |
| 15:18:08 | efried | something about either restarting or not restarting the privsep daemon... | |
| 15:18:12 | dougsz | https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/handlers/main.yml#L211 | |
| 15:18:12 | efried | now I can't remember where I saw it. | |
| 15:18:25 | dougsz | dansmith efried ^ That's the issue afaik | |
| 15:18:26 | dansmith | not restarting is fine, as long as it doesn't lose a reference to the pipe, which I thought was global | |
| 15:19:10 | dansmith | dougsz: oh, right, right, ignore me | |
| 15:19:16 | dansmith | I forgot the rabbit hole went deeper | |
| 15:19:17 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: re-calculate provider mapping during migration https://review.opendev.org/655112 | |
| 15:19:22 | dougsz | np! | |
| 15:19:34 | dansmith | the simple case is that it does a full restart for no reason, but I forgot it also breaks some state as a result | |
| 15:20:41 | efried | someone stephenfin with oslo-core stephenfin should merge that patch | |
| 15:21:19 | stephenfin | whuh? | |
| 15:21:31 | efried | https://review.opendev.org/#/c/641907/ | |
| 15:21:40 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: update allocation in binding profile during migrate https://review.opendev.org/656422 | |
| 15:22:18 | stephenfin | I'm going to finish this cpu-resources reshuffle, but I can take a look after that | |
| 15:22:33 | efried | cool. No hurry, clearly. It's been open since like rocky. | |
| 15:22:49 | efried | sure would be nice to stop having to tell people SIGHUP is broken tho. | |
| 15:22:50 | beekneemech | If you figure it out you'll be ahead of the rest of us. :-) | |
| 15:23:19 | efried | (oh, did I trigger a keyword?) | |
| 15:23:26 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: doc: remove confusing docs about aggregate allocation ratios https://review.opendev.org/678254 | |
| 15:23:48 | beekneemech | Yeah, I added an oslo highlight after the Denver PTL discussion. | |
| 15:24:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extend NeutronFixture to handle migrations https://review.opendev.org/655114 | |
| 15:24:12 | dansmith | beekneemech: efried: I think it might be worth testing that with a real nova | |
| 15:24:13 | dansmith | before just firing it in | |
| 15:24:18 | dansmith | has anyone done that? | |
| 15:24:27 | mriedem | cpu-resources reshuffle is the worst dance move yet | |
| 15:24:37 | efried | dansmith: I ripped it in a devstack, does that count? | |
| 15:24:50 | dansmith | efried: does that mean "tested it" ? | |
| 15:25:09 | mriedem | it means he farted i think | |
| 15:25:27 | efried | I didn't try to hit the "stuff that actually breaks" paths cause I didn't understand them well enough | |
| 15:25:39 | dansmith | then...no | |
| 15:25:42 | efried | but I watched logs through SIGHUP before and after | |
| 15:25:57 | efried | that analysis is in the patch | |
| 15:26:30 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: prepare func test env for moving servers with bandwidth https://review.opendev.org/655109 | |
| 15:26:34 | dansmith | you should be able to validate that before the patch SIGHUP breaks booting instances, and after it doesn't right? | |
| 15:27:41 | efried | dansmith: I'm updating my devstack now, then will try that. | |
| 15:28:22 | mriedem | due to not processing the vif plug event right? you might be waiting 5 minutes if you don't tweak the config | |
| 15:28:37 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.vif_plugging_timeout | |
| 15:28:40 | mriedem | i'd drop that to 60 seconds | |
| 15:28:48 | openstackgerrit | Eric Fried proposed openstack/nova master: Add emulated TPM support to Nova https://review.opendev.org/631363 | |
| 15:28:48 | openstackgerrit | Eric Fried proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | |
| 15:28:52 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate server with ports having resource request https://review.opendev.org/655113 | |
| 15:28:58 | CeeMac | afternoon | |
| 15:29:29 | CeeMac | is there a way to configure nova to use cinder volumes for ephemeral storage and root disks? | |
| 15:29:41 | mriedem | nope | |
| 15:29:55 | CeeMac | that answers that then | |
| 15:29:56 | mriedem | it's come up many a time | |
| 15:30:20 | CeeMac | so, the only option for shared storage backend is to mount the instance disk path to NFS? | |
| 15:30:48 | mriedem | you're talking about different things i think | |
| 15:30:56 | CeeMac | its entirely possible | |
| 15:31:12 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make _rever_allocation nested allocation aware https://review.opendev.org/676138 | |
| 15:31:27 | mriedem | the instance path files are not the same as the root disk - you can have the root disks in rbd but have the instance files (console log, config drive, etc) be on local disk on the compute node | |
| 15:31:29 | mriedem | it gets confusing | |
| 15:31:43 | CeeMac | i don't use ceph, if that helps | |
| 15:31:48 | dansmith | artom: so, what do you think the timetable is for a respin of the numalm set? you said you've got unit tests fixed locally already, so can you do some swizzling of the top two patches and push that up today to get a run started? | |
| 15:31:50 | mriedem | or you could have the instance path files on local disk or shared NFS/rbd, but the root disk is actually a volume in cinder | |
| 15:32:13 | CeeMac | i was thinking in terms of when an instance is launched without specifying to use a cinder volume | |
| 15:32:19 | mriedem | CeeMac: sure, just saying that's a common shared storage backend | |
| 15:32:21 | artom | dansmith, working on updating the func test currently, since confidence seems to be the big thing | |