| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-05 | |||
| 14:01:15 | kaisers | mdbooth: regarding https://review.openstack.org/#/c/554195/ - some time ago you were'nt happy with the issue details, i tried to shed a bit more light on things, could you revisit this at some point pls? | |
| 14:03:29 | mdbooth | kaisers: Is the fix still required in the context of the systemd bugfix? | |
| 14:04:36 | kaisers | mdbooth: yes, the problem ist that we added the --user flag. that fails for multiple reasons | |
| 14:04:52 | kaisers | (i listed them in the commit message and the bugs new description | |
| 14:04:53 | kaisers | ) | |
| 14:06:06 | mdbooth | kaisers: Which version of CentOS doesn't support --user? | |
| 14:06:21 | kaisers | latest as of two days ago | |
| 14:06:22 | mdbooth | Which operating systems don't ship with the systemd fix? | |
| 14:06:33 | mdbooth | kashyap: So CentOS 7? | |
| 14:06:52 | kaisers | The problem ist not the fix for systemd but that it ships with an older systemd that does not support the --user flag | |
| 14:08:29 | kaisers | centos 7 comes with v219, the bug came up with v229 (i think) and was fixed with 23x | |
| 14:09:43 | kaisers | Fixed in v238 according to https://github.com/systemd/systemd/blob/master/NEWS | |
| 14:10:43 | kaisers | mmh, sorry, bug described with v233 in that news link, not v229 | |
| 14:20:20 | openstackgerrit | Chris Dent proposed openstack/nova master: Correct lower-constraints.txt and the related tox job https://review.openstack.org/622972 | |
| 14:20:44 | cdent | stephenfin: ^ has an issue that needs to be resolved before it can be merged, but gets close | |
| 14:25:49 | gibi | stephenfin, mriedem: hi! the last two notification transformation patch is good to go from my perspective https://review.openstack.org/#/q/topic:bp/versioned-notification-transformation-stein+status:open | |
| 14:27:14 | stephenfin | cdent: I'd kill the test, personally. lbragstad added it in 6244a442784 and, while I get what it's doing, I don't think the lower-constraints thing is more of an issue | |
| 14:27:20 | stephenfin | Good to get a second opinion though | |
| 14:28:26 | stephenfin | kaisers: The bug's introduced in v233 and fixed in v238? | |
| 14:29:03 | kaisers | stephenfin: I'm not sure when it was introduced but that's the version for which it is reported in the systemd changelog. | |
| 14:29:13 | kashyap | mdbooth: Hi, was AFK for a break; reading the scroll | |
| 14:29:16 | mriedem | gibi: ack | |
| 14:29:41 | kaisers | stephenfin: But as stated earlier, the more direct problem is that centos 7 comes with v219 and e.g. xenial with 229, both not supporting the --user flag | |
| 14:30:08 | kashyap | mdbooth: Okay, I see kaisers already mentioned which version of systemd ships in CentOS and which one you care about | |
| 14:30:12 | kaisers | stephenfin: scratch that last comment, v229 does support it | |
| 14:30:24 | kaisers | so the old version issue only comes up with centos | |
| 14:30:36 | stephenfin | kaisers: Yup, gotcha | |
| 14:31:05 | mdbooth | mriedem: Hey, about that functional test. Appreciate it made sense to take it out, but I'm trying to work out how we can safely get it back in. Problem is that we basically have no idea why it was failing and now way of finding out that I can think of without running it in CI. Could I perhaps fire up a non-voting job which enabled DEBUG output? | |
| 14:31:36 | mdbooth | I responded on the bug that I *suspect* a performance issue in CI. | |
| 14:31:38 | gibi | mdbooth: does it only fail in CI? | |
| 14:31:59 | mdbooth | gibi: Well it never fails locally. I've run it hundreds of times. | |
| 14:32:12 | stephenfin | kaisers: OK, I can't speak for mdbooth but the last thing I'd like to see is a comment in the code itself explaining that this is only required for CentOS 7 | |
| 14:32:19 | mriedem | mdbooth: i suspect globals | |
| 14:32:20 | gibi | mdbooth: let me try it and see if I can make it fail | |
| 14:32:29 | mriedem | could try the REQUIRES_LOCKING variable or whatever | |
| 14:32:43 | stephenfin | Presumably once CentOS 8 (or 8.1) is out and starts gaining traction, we can remove this again | |
| 14:32:44 | mdbooth | stephenfin kaisers: Also there's a better way to detect systemd | |
| 14:32:48 | mriedem | but i also suggesting writing a libvirt driver extension which overrides methods rather than mock them | |
| 14:32:54 | mriedem | *suggested | |
| 14:32:55 | kaisers | stephenfin: mdbooth: I'm still not sure why the third issue comes up, the --user flag failing when run via oslo_concurrency.processutils.execute. The error shown is similar to running --user as root but the python call should be executed as user nova afaics. I'm still digging into this but currently this fails, too | |
| 14:32:57 | mdbooth | mriedem: What makes you think globals? | |
| 14:33:17 | aspiers | dansmith, mriedem: any thoughts on latest patch set of SEV spec? I tried to capture the results of our Berlin conversations, and sean-k-mooney seems happier now IIUC | |
| 14:33:30 | mriedem | mdbooth: i'd have to look at the bug report again | |
| 14:33:37 | kaisers | mdbooth: regarding the systemd detection i'm all ears, i'm using the method danpb recommended to me some time ago | |
| 14:33:38 | mriedem | in the middle of a different evacuate issue with CI | |
| 14:33:45 | mriedem | aspiers: i haven't looked yet | |
| 14:33:56 | aspiers | mriedem: for your convenience :) https://review.openstack.org/#/c/609779/14..15/specs/stein/approved/amd-sev-libvirt-support.rst | |
| 14:34:38 | kaisers | stephenfin: I'll add the CentOS / systemd release version info to the change | |
| 14:34:51 | stephenfin | gibi: ack. I need to get some reviews done. Only just finished reading 3 weeks of emails (yes, it is already Wednesday :-\) | |
| 14:35:21 | aspiers | mriedem: it looks like only 16MB will need to be accounted for per VM in CONF.reserved_host_memory_mb | |
| 14:35:23 | gibi | stephenfin: no worries, this is the last two patches and we have still time til milestone 2 :) | |
| 14:36:31 | mdbooth | mriedem: Interesting. What difference would it make to override rather than mock? Are you thinking class variables? | |
| 14:36:33 | mriedem | aspiers: and that's something the operator needs to calculate and configure that option based on how many sev instances they think will be on a given host? | |
| 14:36:38 | kaisers | stephenfin: oh, i misread, you do _not_ want that change pointed out to be specifically for centos 7, sry | |
| 14:37:02 | mriedem | mdbooth: i've just seen enough weird mock failures with the libvirt driver to know i don't trust it | |
| 14:37:06 | mriedem | running tests in isolation is fine, | |
| 14:37:17 | aspiers | mriedem: yes, although I suspect in practice that value would be large enough anyway to swallow that amount without extra thought by the operator | |
| 14:37:18 | mriedem | but then running full libvirt driver test suites i've seen weird mock failures | |
| 14:37:37 | mriedem | aspiers: and why can't the driver just report the overhead value if it's a sev-enabled vm? | |
| 14:37:51 | mriedem | *of | |
| 14:38:04 | mriedem | gah | |
| 14:38:08 | aspiers | :) | |
| 14:38:09 | stephenfin | kaisers: No, I do want it | |
| 14:38:31 | kaisers | stephenfin: ok, I'll add | |
| 14:38:39 | stephenfin | kaisers: ta | |
| 14:39:39 | mriedem | aspiers: so yeah, why can't the driver just report the overhead calculation per vm? | |
| 14:40:00 | aspiers | mriedem: it could do that, although as I said in Berlin this is not an increased overhead compared to the non-SEV case | |
| 14:40:03 | mriedem | maybe we won't use it b/c of placement we won't do the ram claim | |
| 14:40:26 | mriedem | aspiers: yeah i still don't understand that, | |
| 14:40:40 | aspiers | mriedem: if you read the latest patch set hopefully all will become clear :) | |
| 14:41:02 | mriedem | i'm very confused by saying, "well operators might need to bump the reserved ram option if the compute hosts these types of instances, but at the same time maybe not because there is no increase in ram usage on the host" | |
| 14:41:37 | aspiers | I've tried very hard to explain this in detail in patchset 15 | |
| 14:41:47 | aspiers | please take a look, and if it's still not clear then let me know | |
| 14:41:52 | dansmith | mriedem: did you see the libvirt statement that the overhead amount is unknowable? | |
| 14:42:02 | mriedem | i've seen that before yes | |
| 14:42:08 | mriedem | but aspiers just said it's about 16MB per Vm | |
| 14:42:25 | aspiers | dansmith: it turns out that it's a more knowable than we thought :) again, patchset goes into great detail on this :) | |
| 14:42:34 | aspiers | *patchset 15 | |
| 14:42:44 | mriedem | right, so, | |
| 14:43:06 | mriedem | at this point, i will likely just +2 out of exhaustion on the spec and we'll push it through and doc the hell out of this in the admin guide | |
| 14:43:11 | mriedem | b/c it's a snowflake feature | |
| 14:43:31 | mriedem | being brutally honest | |
| 14:43:43 | mriedem | not because i'm against it, | |
| 14:43:46 | mriedem | i'm just too dumb to understand it | |
| 14:43:53 | aspiers | haha, I highly doubt that :) | |
| 14:44:01 | mriedem | no really i'm not very smart | |
| 14:44:03 | aspiers | but totally understand that you have limited time for it | |
| 14:44:33 | aspiers | rest assured I will be doc'ing the hell out of it | |
| 14:44:42 | aspiers | that intention is also captured in the spec | |
| 14:45:01 | aspiers | http://logs.openstack.org/79/609779/15/check/openstack-tox-docs/eaf9d54/html/specs/stein/approved/amd-sev-libvirt-support.html#documentation-impact | |
| 14:45:15 | lbragstad | stephenfin cdent hitting issues with I7cd27fe8c39ddfc6ec20f4cfe4d62912d4cebaa4 ? | |
| 14:46:37 | cdent | lbragstad: causes issues in https://review.openstack.org/#/c/622972/ (see commit message) | |
| 14:48:01 | lbragstad | yeah - i don't have a strong attachment to that test... | |
| 14:48:17 | lbragstad | i changed something though so i wanted to test it | |
| 14:48:47 | mriedem | still looking for another core on the allocation ratio data migration patch https://review.openstack.org/#/c/613499/ it's pretty easy | |
| 14:48:53 | mriedem | and we can flush this runway bp today | |
| 14:49:39 | dansmith | yeah, I've had that open since middle of yesterday | |
| 14:51:22 | dansmith | I'll get some protein to go with my caffeine and see if I can make myself do it | |
| 14:53:34 | mriedem | power through | |
| 14:53:44 | mriedem | bauzas: i'm removing reshaper from the runway queue again | |