| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-03 | |||
| 13:46:31 | mriedem | and https://docs.openstack.org/nova/stein/configuration/config.html?highlight=sighup#compute.resource_provider_association_refresh for efried | |
| 13:46:48 | mriedem | ^ might work for that option, but then kills the event listening stuff so you can't boot a server right? | |
| 13:46:56 | mriedem | neutron events i mean | |
| 13:48:21 | mriedem | blech we have stuff in here too https://docs.openstack.org/nova/stein/admin/configuration/schedulers.html?highlight=sighup#compute-capabilities-as-traits | |
| 13:49:59 | mriedem | in other fun news, i learned last night that with cross-cell resize, we have to do a hard delete of the instance in a cell db rather than a soft delete | |
| 13:49:59 | sean-k-mooney | so appraently rechecking a runing job does not work so i can either rebase to head of mater if peopel feel like +w again or i can recheck in 50mins when zuul finishies it current run | |
| 13:50:16 | mriedem | sean-k-mooney: just rebase | |
| 13:50:24 | openstackgerrit | sean mooney proposed openstack/nova master: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649409 | |
| 13:51:20 | sean-k-mooney | done that will retriger the jobs and it just need +w. im going to grab lunch before a meeting so brb | |
| 13:54:40 | mriedem | so i wonder if we could recreate this sighup issue in one of our post-test hook scripts, we'd just sighup the local compute service and then create a server which should timeout waiting for the network-vif-plugged callback right? | |
| 13:54:53 | dansmith | no, | |
| 13:55:02 | dansmith | you have to have a server in the middle of creating when you sighup I think | |
| 13:55:07 | dansmith | before the event comes in | |
| 13:55:38 | dansmith | the other option is probably just to pick another signal as a stop-gap, register for it ourselves, and wire it up to our existing handlers | |
| 13:55:45 | dansmith | but that's even more icky for rc2 | |
| 13:56:04 | mriedem | on a new server create after the signup, won't the _events dict be None and when registering the callback we'd hit this? https://review.openstack.org/#/c/420026/7/nova/compute/manager.py@316 | |
| 13:56:53 | dansmith | I don't think so because we'll be re-created at the point before we get there | |
| 13:57:32 | dansmith | I think it only happens if we end up with a server in the middle of that while the event comes in, but once the sighup has finished the full restart, the new manager is hooked to new rpc connections, etc | |
| 13:57:38 | dansmith | it doesn't completely bork the server forever, AFAIK | |
| 13:57:48 | dansmith | else it would be really obvious that it's totally broken | |
| 13:58:10 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove unreachable codepaths https://review.openstack.org/649559 | |
| 13:59:49 | mriedem | ok i thought the service was fubar after the sighup | |
| 13:59:57 | mriedem | so this is less severe than i thought | |
| 14:01:13 | bauzas | any urgent reviews I should do before RC2 tagging ? | |
| 14:01:33 | dansmith | not that I know of | |
| 14:01:36 | trident | mriedem: Regarding https://review.openstack.org/#/c/648653/ | |
| 14:01:38 | bauzas | https://etherpad.openstack.org/p/nova-stein-rc-potential is pretty done | |
| 14:02:38 | trident | mriedem: You are correct, forbidden traits never worked unless there were also a required trait in the same flavor. If there were, both would be used, if not, the forbidden trait was lost. | |
| 14:03:14 | bauzas | mriedem: https://review.openstack.org/#/c/649454/ wanting it for RC2 ? I can vote on my own change given melwitt proposed it | |
| 14:03:40 | bauzas | I think it should clarify our Stein docs | |
| 14:03:49 | bauzas | even if we don't honestly branch them | |
| 14:06:04 | mriedem | bauzas: i'm waiting to hear that yes we're doing an rc2 | |
| 14:06:15 | mriedem | and then i'd like to step through with the ptl which stein backports are going to go into it | |
| 14:06:28 | bauzas | soooooo... efried? | |
| 14:07:22 | mriedem | there is a 50% chance efried is the PTL coordinating the stein RC2 :) | |
| 14:08:54 | bauzas | oh, right, it's still melwitt's point :p | |
| 14:09:20 | efried | hi, sorry, catching up. | |
| 14:10:03 | mriedem | bauzas: you could start by backporting https://review.openstack.org/#/c/649409/ to stable/stein | |
| 14:10:36 | bauzas | mriedem: ok, I can do it | |
| 14:10:43 | efried | I think we need an RC2 for sure for https://review.openstack.org/#/c/649409/ at least | |
| 14:10:57 | efried | and if we're doing it, we might as well put the docs in, no risk there. | |
| 14:11:21 | mriedem | https://review.openstack.org/#/c/649454/ specifically yeah? | |
| 14:12:00 | efried | yes | |
| 14:13:11 | efried | mriedem: wanna +A that one please? | |
| 14:14:17 | mriedem | done | |
| 14:14:34 | efried | mriedem: Has anyone done docs on the SIGHUP thing? | |
| 14:14:40 | efried | L56 | |
| 14:14:44 | mriedem | no | |
| 14:14:59 | mriedem | i was just creating a devstack env to try and recreate the bug | |
| 14:15:50 | efried | .. note:: SIGHUP is broken, see `bug 1715374`_ | |
| 14:15:50 | efried | mriedem: I can go through and add a quick | |
| 14:15:52 | efried | to all the docs where it's mentioned if you like. | |
| 14:15:52 | openstack | bug 1715374 in OpenStack Compute (nova) "Reloading compute with SIGHUP prevents instances from booting" [High,In progress] https://launchpad.net/bugs/1715374 - Assigned to Ralf Haferkamp (rhafer) | |
| 14:16:57 | mriedem | well, if the nuance is it's broken for a window while servers are being created and waiting for an event when the sighup runs, that gets a bit hard to communicate in all the places we have sighup mentioned in the docs | |
| 14:17:36 | efried | .. note:: SIGHUP behavior is questionable, see `bug 1715374`_ if you try it and things go wobbly. | |
| 14:17:37 | efried | ? | |
| 14:17:37 | openstack | bug 1715374 in OpenStack Compute (nova) "Reloading compute with SIGHUP prevents instances from booting" [High,In progress] https://launchpad.net/bugs/1715374 - Assigned to Ralf Haferkamp (rhafer) | |
| 14:18:28 | mriedem | idk, could give a more detailed explanation of the bug in the reset() portion of https://docs.openstack.org/nova/latest/reference/services.html#the-nova-manager-module for just the compute service | |
| 14:19:06 | mriedem | i wouldn't want to mention a bug in 5 places just to have to remember after the bug is fixed to go back and remove all of those 5 places | |
| 14:20:18 | efried | as you wish | |
| 14:21:00 | openstackgerrit | Sylvain Bauza proposed openstack/nova stable/stein: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649630 | |
| 14:21:09 | bauzas | mriedem: ^ | |
| 14:27:36 | mriedem | is there anything else we want to consider? https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/stein | |
| 14:29:12 | bauzas | mriedem: https://review.openstack.org/#/c/647310/ is controversial for RC2 | |
| 14:31:24 | dansmith | I thought we said no to that? | |
| 14:31:52 | dansmith | and sounds like lyarwood is on board with the original plan too | |
| 14:32:58 | openstackgerrit | Helena proposed openstack/nova-specs master: Spec for a new nova virt driver to manage an RSD, composable infrastructure deployment https://review.openstack.org/648665 | |
| 14:32:59 | lyarwood | yarp, missed that it had already been discussed in the change once I got back online yesterday. gertty-- | |
| 14:34:26 | lyarwood | Does anyone have any idea why nova-stable-maint already has +2 on stable/stein btw? | |
| 14:34:52 | mriedem | i'm not sure if the release team distinguishes anymore | |
| 14:34:54 | mriedem | smcginnis: ^? | |
| 14:36:26 | jaypipes | mriedem, dansmith: for online data migrations, did we envision those migration routines living forever or are we envisioning being able to delete some over time? | |
| 14:36:38 | mriedem | jaypipes: we already have deleted some over time | |
| 14:36:42 | dansmith | jaypipes: we have deleted many | |
| 14:36:50 | jaypipes | oh, ok. | |
| 14:36:55 | mriedem | most recent memory is i removed the request spec and flavor migration ones | |
| 14:37:07 | dansmith | we're not super good at setting that timer and then doing it, like any other cleanup | |
| 14:37:15 | jaypipes | mriedem, dansmith: after looking into mnaser's troubles, I think the removing the keypairs migration entirely might be the best solution. | |
| 14:37:20 | dansmith | but most of them should be relatively no-op-ish if done properly | |
| 14:37:21 | jaypipes | it's Newton-era | |
| 14:37:38 | jaypipes | dansmith: unfortunately that one does a full table scan across instance_extra each time it's run. | |
| 14:37:45 | mriedem | jaypipes: yeah i started looking at that the other day, but i don't think we have a blocker migration or anything in place to make sure the migration has completed before we rip that out | |
| 14:37:47 | mriedem | like we did for flavors | |
| 14:37:47 | jaypipes | dansmith: looking for WHERE keypairs=NULL | |
| 14:38:06 | dansmith | mriedem: we might not be able to for that one | |
| 14:38:19 | jaypipes | dansmith: and for large DBs like mnaser's 3M+ instance_extra records, it really is a resource hog. | |
| 14:38:22 | dansmith | mriedem: but since it's so old, if it hadn't finished, we probably know that other things would be broken | |
| 14:38:24 | mriedem | if there are keypairs in the cell db wouldn't that be an indication? | |
| 14:38:26 | dansmith | jaypipes: I hear you | |
| 14:38:51 | mriedem | for the request spec migration, we didn't have a blocker migration, just a nova-status upgrade check: https://github.com/openstack/nova/commit/ed4fe3ead62c09ec7de7b6a11072295a99997b4f#diff-91e852cb498abb50ca653ef6418bd65a | |
| 14:39:05 | mriedem | oh i added the upgrade check in rocky, dropped the reqspec migration code in stein | |
| 14:39:11 | dansmith | jaypipes: I don't agree that going back to putting this inside the schema migration would make people happy, as his cloud would still be down while he waits for that to happen, but.. you know :) | |
| 14:40:02 | dansmith | jaypipes: hopefully you just meant some sort of version-like sentinel instead of scanning to determine if things needed to be done :) | |
| 14:40:04 | jaypipes | dansmith: why would his cloud be down? | |
| 14:40:14 | dansmith | jaypipes: if we did it in the middle of a schema migration? | |
| 14:40:34 | jaypipes | yeah. | |
| 14:40:52 | dansmith | because he'd be halfway between two schema versions with two versions of code that can't use it? | |
| 14:41:06 | dansmith | i.e. the reason we decoupled those tasks in the first place | |
| 14:41:11 | jaypipes | sorry, you misunderstand... | |
| 14:41:46 | mriedem | wouldn't dropping the keypairs migration be similar to dropping the instance groups migration? https://github.com/openstack/nova/commit/1160921c2d053ce33279ca4ec1f00572271e7c95#diff-91e852cb498abb50ca653ef6418bd65a | |