| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-03 | |||
| 13:41:39 | sean-k-mooney | mriedem: well the patch to fix our lower constratins job just laned so maybe this is a bug that was fixed in later versions of sqlacamy/pymysql | |
| 13:41:54 | mriedem | so after revert there are no allocations for the instance on the target node and the instance has the source node allocations again | |
| 13:42:19 | gibi_off | mriedem: yeah I mixed up it is not moving allocation between hosts it moves between cosnumers | |
| 13:42:35 | mriedem | dansmith: sounds like we might be doing an RC2 for stein, and bug 1715374 is latent, but do you think it would be worth putting out a known issue release note for stein anyway? | |
| 13:42:36 | openstack | bug 1715374 in OpenStack Compute (nova) "Reloading compute with SIGHUP prevents instances from booting" [High,In progress] https://launchpad.net/bugs/1715374 - Assigned to Ralf Haferkamp (rhafer) | |
| 13:42:59 | dansmith | mriedem: I dunno, it's been this way for a long time apparently | |
| 13:43:06 | dansmith | so doesn't really seem like it | |
| 13:43:38 | gibi_off | mriedem: anyhow I think the warning can be deleted | |
| 13:43:53 | mriedem | gibi_off: or at least dropped to debug | |
| 13:44:03 | gibi_off | yeah | |
| 13:45:31 | mriedem | dansmith: just thinking about our upgrade docs and such that mention to use sighup during an upgrade https://docs.openstack.org/nova/stein/user/upgrade.html?highlight=sighup#concepts | |
| 13:45:51 | dansmith | well, I know | |
| 13:45:52 | mriedem | maybe that should have a note instead | |
| 13:46:01 | dansmith | yeah, that would make more sense I think | |
| 13:46:31 | mriedem | and https://docs.openstack.org/nova/stein/configuration/config.html?highlight=sighup#compute.resource_provider_association_refresh for efried | |
| 13:46:48 | mriedem | ^ might work for that option, but then kills the event listening stuff so you can't boot a server right? | |
| 13:46:56 | mriedem | neutron events i mean | |
| 13:48:21 | mriedem | blech we have stuff in here too https://docs.openstack.org/nova/stein/admin/configuration/schedulers.html?highlight=sighup#compute-capabilities-as-traits | |
| 13:49:59 | sean-k-mooney | so appraently rechecking a runing job does not work so i can either rebase to head of mater if peopel feel like +w again or i can recheck in 50mins when zuul finishies it current run | |
| 13:49:59 | mriedem | in other fun news, i learned last night that with cross-cell resize, we have to do a hard delete of the instance in a cell db rather than a soft delete | |
| 13:50:16 | mriedem | sean-k-mooney: just rebase | |
| 13:50:24 | openstackgerrit | sean mooney proposed openstack/nova master: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649409 | |
| 13:51:20 | sean-k-mooney | done that will retriger the jobs and it just need +w. im going to grab lunch before a meeting so brb | |
| 13:54:40 | mriedem | so i wonder if we could recreate this sighup issue in one of our post-test hook scripts, we'd just sighup the local compute service and then create a server which should timeout waiting for the network-vif-plugged callback right? | |
| 13:54:53 | dansmith | no, | |
| 13:55:02 | dansmith | you have to have a server in the middle of creating when you sighup I think | |
| 13:55:07 | dansmith | before the event comes in | |
| 13:55:38 | dansmith | the other option is probably just to pick another signal as a stop-gap, register for it ourselves, and wire it up to our existing handlers | |
| 13:55:45 | dansmith | but that's even more icky for rc2 | |
| 13:56:04 | mriedem | on a new server create after the signup, won't the _events dict be None and when registering the callback we'd hit this? https://review.openstack.org/#/c/420026/7/nova/compute/manager.py@316 | |
| 13:56:53 | dansmith | I don't think so because we'll be re-created at the point before we get there | |
| 13:57:32 | dansmith | I think it only happens if we end up with a server in the middle of that while the event comes in, but once the sighup has finished the full restart, the new manager is hooked to new rpc connections, etc | |
| 13:57:38 | dansmith | it doesn't completely bork the server forever, AFAIK | |
| 13:57:48 | dansmith | else it would be really obvious that it's totally broken | |
| 13:58:10 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove unreachable codepaths https://review.openstack.org/649559 | |
| 13:59:49 | mriedem | ok i thought the service was fubar after the sighup | |
| 13:59:57 | mriedem | so this is less severe than i thought | |
| 14:01:13 | bauzas | any urgent reviews I should do before RC2 tagging ? | |
| 14:01:33 | dansmith | not that I know of | |
| 14:01:36 | trident | mriedem: Regarding https://review.openstack.org/#/c/648653/ | |
| 14:01:38 | bauzas | https://etherpad.openstack.org/p/nova-stein-rc-potential is pretty done | |
| 14:02:38 | trident | mriedem: You are correct, forbidden traits never worked unless there were also a required trait in the same flavor. If there were, both would be used, if not, the forbidden trait was lost. | |
| 14:03:14 | bauzas | mriedem: https://review.openstack.org/#/c/649454/ wanting it for RC2 ? I can vote on my own change given melwitt proposed it | |
| 14:03:40 | bauzas | I think it should clarify our Stein docs | |
| 14:03:49 | bauzas | even if we don't honestly branch them | |
| 14:06:04 | mriedem | bauzas: i'm waiting to hear that yes we're doing an rc2 | |
| 14:06:15 | mriedem | and then i'd like to step through with the ptl which stein backports are going to go into it | |
| 14:06:28 | bauzas | soooooo... efried? | |
| 14:07:22 | mriedem | there is a 50% chance efried is the PTL coordinating the stein RC2 :) | |
| 14:08:54 | bauzas | oh, right, it's still melwitt's point :p | |
| 14:09:20 | efried | hi, sorry, catching up. | |
| 14:10:03 | mriedem | bauzas: you could start by backporting https://review.openstack.org/#/c/649409/ to stable/stein | |
| 14:10:36 | bauzas | mriedem: ok, I can do it | |
| 14:10:43 | efried | I think we need an RC2 for sure for https://review.openstack.org/#/c/649409/ at least | |
| 14:10:57 | efried | and if we're doing it, we might as well put the docs in, no risk there. | |
| 14:11:21 | mriedem | https://review.openstack.org/#/c/649454/ specifically yeah? | |
| 14:12:00 | efried | yes | |
| 14:13:11 | efried | mriedem: wanna +A that one please? | |
| 14:14:17 | mriedem | done | |
| 14:14:34 | efried | mriedem: Has anyone done docs on the SIGHUP thing? | |
| 14:14:40 | efried | L56 | |
| 14:14:44 | mriedem | no | |
| 14:14:59 | mriedem | i was just creating a devstack env to try and recreate the bug | |
| 14:15:50 | efried | mriedem: I can go through and add a quick | |
| 14:15:50 | efried | .. note:: SIGHUP is broken, see `bug 1715374`_ | |
| 14:15:52 | openstack | bug 1715374 in OpenStack Compute (nova) "Reloading compute with SIGHUP prevents instances from booting" [High,In progress] https://launchpad.net/bugs/1715374 - Assigned to Ralf Haferkamp (rhafer) | |
| 14:15:52 | efried | to all the docs where it's mentioned if you like. | |
| 14:16:57 | mriedem | well, if the nuance is it's broken for a window while servers are being created and waiting for an event when the sighup runs, that gets a bit hard to communicate in all the places we have sighup mentioned in the docs | |
| 14:17:36 | efried | .. note:: SIGHUP behavior is questionable, see `bug 1715374`_ if you try it and things go wobbly. | |
| 14:17:37 | openstack | bug 1715374 in OpenStack Compute (nova) "Reloading compute with SIGHUP prevents instances from booting" [High,In progress] https://launchpad.net/bugs/1715374 - Assigned to Ralf Haferkamp (rhafer) | |
| 14:17:37 | efried | ? | |
| 14:18:28 | mriedem | idk, could give a more detailed explanation of the bug in the reset() portion of https://docs.openstack.org/nova/latest/reference/services.html#the-nova-manager-module for just the compute service | |
| 14:19:06 | mriedem | i wouldn't want to mention a bug in 5 places just to have to remember after the bug is fixed to go back and remove all of those 5 places | |
| 14:20:18 | efried | as you wish | |
| 14:21:00 | openstackgerrit | Sylvain Bauza proposed openstack/nova stable/stein: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649630 | |
| 14:21:09 | bauzas | mriedem: ^ | |
| 14:27:36 | mriedem | is there anything else we want to consider? https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/stein | |
| 14:29:12 | bauzas | mriedem: https://review.openstack.org/#/c/647310/ is controversial for RC2 | |
| 14:31:24 | dansmith | I thought we said no to that? | |
| 14:31:52 | dansmith | and sounds like lyarwood is on board with the original plan too | |
| 14:32:58 | openstackgerrit | Helena proposed openstack/nova-specs master: Spec for a new nova virt driver to manage an RSD, composable infrastructure deployment https://review.openstack.org/648665 | |
| 14:32:59 | lyarwood | yarp, missed that it had already been discussed in the change once I got back online yesterday. gertty-- | |
| 14:34:26 | lyarwood | Does anyone have any idea why nova-stable-maint already has +2 on stable/stein btw? | |
| 14:34:52 | mriedem | i'm not sure if the release team distinguishes anymore | |
| 14:34:54 | mriedem | smcginnis: ^? | |
| 14:36:26 | jaypipes | mriedem, dansmith: for online data migrations, did we envision those migration routines living forever or are we envisioning being able to delete some over time? | |
| 14:36:38 | mriedem | jaypipes: we already have deleted some over time | |
| 14:36:42 | dansmith | jaypipes: we have deleted many | |
| 14:36:50 | jaypipes | oh, ok. | |
| 14:36:55 | mriedem | most recent memory is i removed the request spec and flavor migration ones | |
| 14:37:07 | dansmith | we're not super good at setting that timer and then doing it, like any other cleanup | |
| 14:37:15 | jaypipes | mriedem, dansmith: after looking into mnaser's troubles, I think the removing the keypairs migration entirely might be the best solution. | |
| 14:37:20 | dansmith | but most of them should be relatively no-op-ish if done properly | |
| 14:37:21 | jaypipes | it's Newton-era | |
| 14:37:38 | jaypipes | dansmith: unfortunately that one does a full table scan across instance_extra each time it's run. | |
| 14:37:45 | mriedem | jaypipes: yeah i started looking at that the other day, but i don't think we have a blocker migration or anything in place to make sure the migration has completed before we rip that out | |
| 14:37:47 | jaypipes | dansmith: looking for WHERE keypairs=NULL | |
| 14:37:47 | mriedem | like we did for flavors | |
| 14:38:06 | dansmith | mriedem: we might not be able to for that one | |
| 14:38:19 | jaypipes | dansmith: and for large DBs like mnaser's 3M+ instance_extra records, it really is a resource hog. | |