| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-02 | |||
| 15:55:00 | sean-k-mooney | tcp is expressed not recommended for production use which is why im uneasey about that being our defualt | |
| 15:55:24 | stephenfin | sean-k-mooney: I can do that so but it'll warrant an Upgrade reno. ssh or tls as the default? | |
| 15:55:27 | sean-k-mooney | stephenfin: kolla uses ssh not sure about tripplo | |
| 15:55:48 | stephenfin | I think TripleO uses TLS if TLS everywhere is toggled but owalsh would need to confirm | |
| 15:55:52 | artom | gryf, around? You wanted to ask about NUMA live migration? | |
| 15:56:16 | sean-k-mooney | stephenfin: i would go with ssh but it likely somthing we want to discuss perhaps in the nova team meeting | |
| 15:56:21 | owalsh | stephenfin, sean-k-mooney: yea, TLS if it's enabled, otherwise ssh tunnelling | |
| 15:56:24 | sean-k-mooney | or as a bug/bluepint | |
| 15:59:10 | sean-k-mooney | stephenfin: if we dont change the default it would be nice to atleast issue a wanrning on compute agent startup so that operators are very ware that they should not be using tcp by defualt | |
| 16:00:18 | owalsh | sean-k-mooney: I wouldn't be surprised if it got #ifdef-ed out of libvirt by default too | |
| 16:00:37 | efried | mdbooth: Oh, I meant to mention this earlier... | |
| 16:01:18 | efried | we don't need context.nested, because that's implemented natively in py2 (I think as of 2.6?) and py3. | |
| 16:01:28 | sean-k-mooney | owalsh: its compiled into the centos bin but its disabled in the libvirt deamon config so you have to expcily opt in to it | |
| 16:01:49 | efried | mdbooth: s/context.nested/nested_contexts/ | |
| 16:02:16 | mdbooth | efried: It's in py3? Where? | |
| 16:02:27 | efried | mdbooth: natively | |
| 16:02:31 | owalsh | sean-k-mooney: yes, but all of the random blogs on setting up migration enabled it | |
| 16:02:33 | mdbooth | Called what? | |
| 16:02:43 | efried | mdbooth: Called 'with' :) | |
| 16:02:48 | mdbooth | efried: Oh, you mean the 'with-syntax' | |
| 16:02:54 | efried | mdbooth: maybe I'm not understanding what you're trying to do with it. | |
| 16:02:56 | sean-k-mooney | efried: really when did that get backported | |
| 16:02:57 | mdbooth | Right, but that's *not* implemented in py2 | |
| 16:03:02 | mdbooth | The optional bit, that is | |
| 16:03:18 | sean-k-mooney | efried: we still have at least 1 place where we stub out compatiably in nova | |
| 16:03:20 | mdbooth | You can't say: | |
| 16:03:31 | mdbooth | with X, Y (unless Y is None) | |
| 16:03:32 | mdbooth | : | |
| 16:04:16 | mdbooth | Basically the choices for optional composition are contextlib.nested, and ExitStack, and neither is available in both py2 and py3 | |
| 16:04:31 | efried | mdbooth: oh, I was missing the filter | |
| 16:04:37 | stephenfin | mdbooth: https://pythonclock.org/ | |
| 16:05:12 | mdbooth | stephenfin: Python 3 is the culprit here, because Python 2 was already here. | |
| 16:05:43 | mdbooth | Deprecating a whole programming language was a monumentally stupid idea. | |
| 16:06:02 | openstackgerrit | Elod Illes proposed openstack/nova stable/ocata: [Stable Only] Add amd-ssbd and amd-no-ssb CPU flags https://review.openstack.org/607296 | |
| 16:09:12 | mdbooth | efried: Incidentally, you can make Y a 'null context', i.e. a context manager which does nothing other than execute whatever it's given. | |
| 16:09:25 | mdbooth | And py3 includes a contextlib.nullcontext | |
| 16:09:29 | mdbooth | But py2 doesn't | |
| 16:09:44 | mdbooth | So I could have written one, but it would still be duplicating code in py3 | |
| 16:10:32 | sean-k-mooney | mdbooth: it could jsut delegate to the py3 verions on py3 and implement the 10 lines it takes on py2. | |
| 16:10:42 | efried | or after the import: | |
| 16:10:43 | efried | ? | |
| 16:10:43 | efried | contextlib.nullcontext = lambda x: x | |
| 16:10:43 | efried | if not hasattr(contextlib, 'nullcontext'): | |
| 16:10:57 | sean-k-mooney | we cant drop py2 support for 2 more releases so i dont see an issue with it | |
| 16:10:58 | efried | anyway, what you've done is foyne. | |
| 16:11:10 | mdbooth | efried: That's not quite a nullcontext. You probably need at least 4 lines for it | |
| 16:11:16 | efried | yeah yeah | |
| 16:11:49 | mdbooth | efried: The point is there were a bunch of ways to do it, but none of them are as good as what you could do if we were only targetting a single programming language. | |
| 16:12:09 | efried | yup, I get it. You had to choose among several ugly options. What you've done is foyne. | |
| 16:15:07 | artom | Weird - I'm trying to reproduce a bug where if an instance reschedules it gets two vifs - I'm trying to force a reschedule by just raising in build_and_run_instance one one of the two computes. Except *no instance ever lands on that compute* | |
| 16:15:54 | artom | That host definitely doesn't get filtered out, it's enabled, etc | |
| 16:15:56 | openstackgerrit | Matthew Booth proposed openstack/nova master: DNM: Add some debug logging https://review.openstack.org/607301 | |
| 16:18:26 | sean-k-mooney | artom: try raising right at the point the libvirt driver calls libvir to spawn the instance | |
| 16:18:52 | gibi | mriedem: hi, do you have something for the notification subteam meeting? I don't have any | |
| 16:19:14 | artom | sean-k-mooney, it's not even that. I've booted 10 instances with --min-count, and they've *all* gone to the "wrong" host | |
| 16:21:02 | sean-k-mooney | artom: you didnt hit the auto host down on faiure codepath? | |
| 16:21:33 | artom | sean-k-mooney, well, how would I see that? The hypervisor shows as up, same for the service | |
| 16:23:39 | dansmith | artom: there's a weigher now | |
| 16:23:48 | dansmith | for the fail count, | |
| 16:24:02 | dansmith | but that will only take effect if you have failed once on that compute already | |
| 16:24:39 | artom | dansmith, aha, thank you! | |
| 16:24:52 | artom | That would explain it - now, presumably this fail count is hidden in the DB somewhere. | |
| 16:25:29 | sean-k-mooney | dansmith: didnt you also add something last cycle to auto makre the state as disabled after 10 ish failed boots in addtion to the weigher | |
| 16:25:46 | dansmith | sean-k-mooney: no, the weigher replaced the auto-down thing | |
| 16:25:53 | sean-k-mooney | artom: im assuming openstack compute service list shoul the state as up and statuse as enabled | |
| 16:26:29 | sean-k-mooney | dansmith: ah ok i guess that makes sense we had no auto up right so you had to manully go mark it up | |
| 16:27:24 | dansmith | right | |
| 16:27:25 | artom | dansmith, hrmm, does the fail count get reset on service restart? | |
| 16:27:55 | artom | Btw, this would be a think we should probably expose in the hypervisors API or something | |
| 16:28:03 | sean-k-mooney | artom: if its in memory you would have to restart the schduler | |
| 16:28:13 | dansmith | it's in the host state | |
| 16:28:23 | mriedem | gibi: no, although i do have something i can stick in your ear... | |
| 16:28:24 | dansmith | so we should pull that each time in the filter scheduler, IIRC | |
| 16:29:12 | mriedem | gibi: https://review.openstack.org/#/c/603930/6/nova/conductor/tasks/cross_cell_migrate.py@519 for the cross-cell resize stuff i'm adding a new conductor task and the conductor code relies on that old scheduler_utils.set_vm_state_and_notify method, which sends an error notification but those are all legacy | |
| 16:29:35 | mriedem | https://github.com/openstack/nova/blob/cce3208cc28268e4b50e155c205bcab9f1da2a4b/nova/scheduler/utils.py#L600 | |
| 16:29:44 | sean-k-mooney | artom: this is the filter https://github.com/openstack/nova/blob/master/nova/scheduler/weights/compute.py#L24 you could just set the weight multipler to 0 to disable it | |
| 16:29:47 | mriedem | i'm not entirely sure how to make that generic thing useful for versioned notifications | |
| 16:29:48 | dansmith | artom: you can just zero or invert the weight and see if that changes it | |
| 16:29:59 | gibi | mriedem: looking | |
| 16:30:18 | mriedem | artom: see compute_nodes.stats column | |
| 16:30:26 | mriedem | which is a serialized dict i think.... | |
| 16:30:34 | artom | mriedem, saw it, it's a JSON blob | |
| 16:30:38 | mriedem | yup | |
| 16:30:53 | mriedem | not a terrible idea to report that out of the os-hypervisors API | |
| 16:30:59 | artom | Restarting sched and/or compute seems to do what I want (ie, effectively reset the failure count) | |
| 16:31:10 | artom | mriedem, yeah eh? It would avoid more losers like me getting lost | |
| 16:31:36 | gibi | mriedem: there is an open (and old) patch about that missing set of notifications https://review.openstack.org/#/c/482629/9/nova/scheduler/utils.py | |
| 16:32:10 | gibi | mriedem: s/old/not reviewed by me for a while/ | |
| 16:32:12 | mriedem | oh boy...that's a big one | |
| 16:32:48 | gibi | mriedem: stephenfin was happy with that not so long ago | |
| 16:33:35 | gibi | mriedem: anyhow I made a note to review this sooner than later | |
| 16:33:51 | mriedem | definitely not a high priority, but thanks for noting this exists | |
| 16:35:13 | gibi | mriedem: the burndown chart remembered that it exists not me :) | |
| 16:36:21 | gibi | mriedem: anyhow I will not run the notification subteam meeting this week either then | |
| 16:39:42 | mriedem | gibi: ok | |
| 16:39:55 | mriedem | gibi: do you want to just cancel the meeting? | |
| 16:40:07 | mriedem | i mean, indefinitely | |
| 16:41:34 | gibi | mriedem: that decision is forming slowly in my mind in the past weeks | |
| 16:41:46 | gibi | mriedem: I think it would make sense to cancel it indefinitely | |
| 16:44:32 | mriedem | go for it, | |