| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-03 | |||
| 15:05:29 | dansmith | we probably need to nuke all those "added in neutron" migrations | |
| 15:07:04 | mriedem | i've got a local change, sec | |
| 15:07:11 | dansmith | only one more now I guess | |
| 15:07:20 | dansmith | looking at your patch from earlier there were abunch | |
| 15:07:25 | mriedem | yeah https://review.openstack.org/#/q/topic:remove-newton-online-compat-code+(status:open+OR+status:merged) | |
| 15:07:29 | mriedem | i tried | |
| 15:07:30 | mriedem | got tired | |
| 15:22:39 | cdent | Is there a concise description on the rules about migrations between AZs somewhere? cold and live, force and not force? | |
| 15:23:15 | mriedem | likely not, at least off the top of my head, but i could probably explain it quick | |
| 15:23:26 | mriedem | then we could report a docs bug to fill that in later | |
| 15:24:22 | cdent | I'll take that | |
| 15:24:29 | mriedem | tl;dr if the user creates the server with a specific AZ or https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.default_schedule_zone is not None (meaning it goes into some default AZ), then the instance is restricted to that AZ for all move operations, | |
| 15:24:38 | mriedem | UNLESS it is forced like with live migrate or evacuate | |
| 15:25:17 | mriedem | which, by default, with OSC you get a forced live migration every time | |
| 15:25:25 | mriedem | so very easy to shoot yourself in the foot there as an admin | |
| 15:25:42 | mriedem | which is why there are at least 5 patches to osc to fix that, listed at L18 here https://etherpad.openstack.org/p/DEN-osc-compute-api-gaps | |
| 15:26:00 | cdent | \o/ | |
| 15:26:56 | cdent | are there any restrictions that distinguish the default AZ as special with regard to other AZs? | |
| 15:27:05 | mriedem | docs on migrate caveats regarding AZs could probably live here https://docs.openstack.org/nova/latest/user/aggregates.html#availability-zones-azs if you want to report a bug and i can wordsmith it later | |
| 15:27:11 | mriedem | no | |
| 15:27:31 | mriedem | if DEFAULT.default_schedule_zone is not None and the user doesn't request an AZ explicitly, it's treated as if the user did request the default AZ | |
| 15:27:43 | cdent | I guess if you specify that a sever is in foo-AZ (non-default) then you can't cold migrate from there to any other `one? | |
| 15:28:06 | mriedem | that happens here https://github.com/openstack/nova/blob/357da989c194a8b59842629cb64b2809143a4eae/nova/api/openstack/compute/servers.py#L641 | |
| 15:28:19 | mriedem | cdent: correct | |
| 15:28:40 | cdent | okay. thanks. I'll make a bug, summarize this stuff there. | |
| 15:28:48 | mriedem | note there is a spec proposed to allow pasing a new AZ on unshelve, which i think is reasonable | |
| 15:30:11 | mriedem | [cinder]/cross_az_attach=False also makes this all more complicated.... | |
| 15:30:35 | mriedem | ala https://review.openstack.org/#/c/469675/ wee | |
| 15:31:10 | mriedem | tl;dr if the cloud is configured for [cinder]/cross_az_attach=False and you boot from volume where the volume is in a non-default zone, server create explodes immediately | |
| 15:31:27 | mriedem | *and you don't create the server in the same zone | |
| 15:31:36 | mriedem | sorrison suffers from ^ | |
| 15:32:05 | mriedem | side effects include chronic vegimitis | |
| 15:32:12 | dansmith | gdi mriedem, save your depressing stuff for mondays will you? wednesday is supposed to be cresting the hill of depression and heading down to friday happiness | |
| 15:32:54 | mriedem | being constantly depressed about bugs from essex still being in our code is all i have to live for anymore | |
| 15:34:37 | mriedem | not to mention all of my software skills are stuck in 6 years ago and i'm not learning anything new as a developer...wahwah | |
| 15:35:59 | artom | You're learning new soft skills | |
| 15:36:08 | artom | Like... how to deal with depression ^_^ | |
| 15:37:38 | dansmith | nice | |
| 15:40:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop migrate_keypairs_to_api_db data migration https://review.openstack.org/649648 | |
| 15:40:48 | mriedem | that'll be good on my next interview | |
| 15:40:58 | mriedem | dansmith: jaypipes: mnaser: ^ there you go | |
| 15:43:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop migrate_keypairs_to_api_db data migration https://review.openstack.org/649648 | |
| 15:45:42 | mriedem | cdent: heh sound familiar? https://bugs.launchpad.net/nova/+bug/1822986 | |
| 15:45:43 | openstack | Launchpad bug 1822986 in OpenStack Compute (nova) "Not clear if www_authenticate_uri is really needed" [Undecided,New] | |
| 15:45:55 | cdent | oh hai | |
| 15:46:20 | cdent | hmm, yes, rather familiar | |
| 15:51:09 | melwitt | mriedem, efried: I thought I was coordinating RC2, but if someone else wants to, that's fine with me. I put the things I thought should go into it on https://etherpad.openstack.org/p/nova-stein-rc-potential last night | |
| 15:52:44 | mriedem | melwitt: i think those are the same two changes that are being put in, so it's aligned | |
| 15:53:14 | melwitt | ok, cool | |
| 16:00:01 | jaypipes | mriedem: ++ | |
| 16:06:09 | melwitt | mriedem, efried: fyi I've proposed RC2 at https://review.openstack.org/649656 | |
| 16:19:00 | efried | mriedem: was away for a bit there. Did you decide to do something about SIGHUP for RC2? | |
| 16:19:55 | mriedem | efried: no | |
| 16:20:05 | mriedem | i went on a online data migrations tangent | |
| 16:20:24 | efried | ight | |
| 16:21:23 | mriedem | kashyap: MIN_LIBVIRT_VERSION = (3, 0, 0) and MIN_LIBVIRT_POSTCOPY_VERSION = (1, 3, 3), do you have a change to remove MIN_LIBVIRT_POSTCOPY_VERSION? | |
| 16:24:24 | mriedem | doesn't look like it | |
| 16:34:43 | mriedem | NewBruce: what do you have set for live_migration_permit_post_copy in nova.conf on your compute services? | |
| 16:38:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: drop MIN_LIBVIRT_POSTCOPY_VERSION https://review.openstack.org/649671 | |
| 16:50:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: remove conditional on VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY https://review.openstack.org/649674 | |
| 16:52:07 | mriedem | sean-k-mooney: so on https://review.openstack.org/#/c/649464/ we're never getting a post-copy event during live migration so we don't activate the dest host port binding prior to post_live_migration_at_destination, so that kind of throws that theory out the window as being the issue NewBruce is hitting | |
| 16:52:13 | openstackgerrit | Eric Fried proposed openstack/nova stable/rocky: Fix bug preventing forbidden traits from working https://review.openstack.org/649603 | |
| 16:58:49 | sean-k-mooney | mriedem: ya i noticed that this morning that it passed | |
| 17:00:07 | sean-k-mooney | i was wondering if we shoudl maybe try and force the compute level to be different on the contoler vs the compute and see if that triggers the issue | |
| 17:00:53 | sean-k-mooney | mriedem: were you going to resubmit with post copy enabled? | |
| 17:02:16 | mriedem | sean-k-mooney: yeah i was going to try enabling post-copy but these cirros guests are so tiny i'm not sure it will mean anything | |
| 17:02:37 | mriedem | efried: dansmith: on that SIGHUP n-cpu issue, i tried that in a train devstack created this morning and i don't even get to the wait for network event part | |
| 17:02:38 | mriedem | http://paste.openstack.org/show/748820/ | |
| 17:02:54 | mriedem | libvirt blows up in privsep | |
| 17:03:16 | sean-k-mooney | mriedem: i might try and create a neutron fullstack test to reproduce the nutron db error | |
| 17:03:21 | mriedem | this is how i hup'ed: sudo systemctl kill -s HUP devstack@n-cpu.service | |
| 17:03:27 | dansmith | mriedem: what happens if you try it again? | |
| 17:03:37 | mriedem | the server create? or the hup? | |
| 17:03:42 | dansmith | mriedem: the create | |
| 17:03:55 | mriedem | same thing | |
| 17:03:58 | dansmith | okay | |
| 17:04:06 | dansmith | I have not seen it manifest that way | |
| 17:04:26 | dansmith | obviously that seems like a much bigger deal | |
| 17:04:36 | mriedem | unplug on cleaning up from the failure also then blows up b/c privsep http://paste.openstack.org/show/748821/ | |
| 17:04:48 | dansmith | which should be more evidence that the oslo behavior is completely wrong | |
| 17:06:10 | mriedem | did a full systemctl restart and created a server and it was fine as expected | |
| 17:06:24 | dansmith | I forget, but can people still choose to run privsep things via rootwrap so we can restart the daemon? | |
| 17:06:36 | dansmith | because if so, maybe that's why you see that and it's devstack default config or whatever | |
| 17:06:56 | dansmith | maybe I could get mikal to yell at us over twitter to answer that | |
| 17:07:25 | mriedem | idk, privsep config et al is something i haven't had to look at in years | |
| 17:07:35 | mriedem | but was always confusing to me | |
| 17:07:37 | dansmith | yeah | |
| 17:07:55 | dansmith | or maybe it's different with some of the privsep things we merged recently, | |
| 17:08:05 | dansmith | but people reporting it are missing more of the privsepification | |
| 17:08:44 | dansmith | did you create a server before the hup too? | |
| 17:12:13 | mriedem | yup | |
| 17:12:35 | mriedem | created test1, fine, sighup'ed, create test2 fails, create test3 fails, restart n-cpu, create test4 ok | |
| 17:14:33 | mriedem | commented 34 and 35 fwiw in bug https://bugs.launchpad.net/nova/+bug/1715374 | |
| 17:14:34 | openstack | Launchpad bug 1715374 in OpenStack Compute (nova) "Reloading compute with SIGHUP prevents instances from booting" [High,In progress] - Assigned to Ralf Haferkamp (rhafer) | |
| 17:14:52 | dansmith | okay just trying to think of reasons you see it differently than reported, but probably because of recent changes I guess | |
| 17:18:21 | mriedem | the privsep-helper child processes are definitely gone after the SIGHUP http://paste.openstack.org/show/748822/ | |
| 17:19:26 | dansmith | so we probably just didn't notice before or something as fewer things used it | |
| 17:19:37 | dansmith | but I don't think there's any reason that they're being killed now and not before | |
| 17:20:48 | mriedem | yeah, this is what i see in the n-cpu logs on the HUP http://paste.openstack.org/show/748823/ | |
| 17:21:08 | mriedem | note the | |