| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-06 | |||
| 13:47:51 | mriedem | so it would be rhosp specific? | |
| 13:48:04 | mriedem | or python-libvirt patches, something external to nova? | |
| 13:48:15 | dansmith | mriedem: I doubt it, but if they did and it was related, then they'd have been fixed by this (if that's what you mean) | |
| 13:48:21 | mriedem | also, if we land this on master, will backports be proposed upstream? | |
| 13:48:31 | dansmith | mriedem: they're probably on newer *libvirt*, but I dunno how often python-libvirt changes | |
| 13:49:01 | mriedem | i think you have to rebuild python-libvirt from whatever version of libvirt binary you're using | |
| 13:49:05 | mriedem | otherwise things get weird | |
| 13:49:32 | dansmith | mriedem: I dunno, ask mdbooth. I would expect he would want to | |
| 13:49:49 | mriedem | i remember we had issues in the past with running libvirt x in the gate jobs but python-libvirt was built against y and reporting y to nova, so we'd try to use features that weren't actually there | |
| 13:49:50 | dansmith | mriedem: right but that's just because the bindings are auto-generated I think | |
| 13:50:10 | mriedem | ok, so i'm less comfortable with backporting a thing that we're not sure fixes the issue | |
| 13:50:25 | mriedem | it's one thing to land on master and let it simmer until people get to it, | |
| 13:50:32 | mriedem | than to backport it and force it on people with otherwise stable environments | |
| 13:50:40 | dansmith | although we are right before FF, so not as much runway | |
| 13:51:01 | mriedem | rh probably just cares that it's merged on master and can backport internally is less of an issue yeah? | |
| 13:51:19 | mriedem | merge conflict risk on stable branches and such i know | |
| 13:51:24 | dansmith | maybe we could get efried to comment on the risk, and maybe cdent on the actual eventlet bits? | |
| 13:51:32 | dansmith | I'm not sure what context cdent has on eventlet, but it seems like sometimes he has a lot | |
| 13:51:44 | cdent | hello what? | |
| 13:51:49 | mriedem | efried isn't a stable branch maintainer though, or you just mean risk for FF? | |
| 13:51:49 | cdent | which thing? | |
| 13:52:01 | dansmith | mriedem: I mean for merging to master right now or in general yeah | |
| 13:52:10 | mriedem | cdent: https://review.opendev.org/#/c/677736/ | |
| 13:52:20 | mriedem | cdent: tl;dr old ass code, | |
| 13:52:40 | mriedem | suddenly 2 rhosp customers are seeing weird issues which mdbooth thinks are related to libvirt + python + eventlet, | |
| 13:52:41 | dansmith | (I wish I was working on things that *have* to merge by FF right now) | |
| 13:52:52 | mriedem | but there isn't a clear indication that patch fixes those problems | |
| 13:52:54 | openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | |
| 13:53:07 | aspiers | stephenfin: Your doc feedback should be addressed now ^^^ | |
| 13:53:16 | dansmith | mriedem: I see no indication that it fixes those problems.. I think the only indication is that it "seems wrong" and "these tests pass" | |
| 13:53:36 | mriedem | sure, so ruling it out i guess | |
| 13:53:53 | mriedem | i also need to investigate these fixes that are blocking the ironic gate atm | |
| 13:54:01 | mriedem | since i'm guessing ironic would like to merge code before FF | |
| 13:54:15 | aspiers | stephenfin: https://review.opendev.org/#/c/666616/47..48/nova/conf/libvirt.py | |
| 13:54:18 | donnyd | sean-k-mooney: Is the NUMA job still throwing errors? | |
| 13:54:34 | aspiers | stephenfin: BTW I've checked that the rst all renders OK | |
| 13:54:54 | sean-k-mooney | donnyd: i have not re run it | |
| 13:55:08 | sean-k-mooney | with the queue thats in the gate i was going to wiat until the weekend | |
| 13:55:23 | sean-k-mooney | donnyd: i can kick it off now if you like and see | |
| 13:55:30 | donnyd | sure | |
| 13:58:07 | sean-k-mooney | donnyd: there is a build stitting in the experimental pipeline now | |
| 13:58:16 | sean-k-mooney | donnyd: ill let you know how it goes | |
| 13:58:20 | donnyd | kk | |
| 13:58:29 | donnyd | where can i check its status? | |
| 13:58:39 | sean-k-mooney | http://zuul.openstack.org/status | |
| 13:58:47 | sean-k-mooney | filter by 679656 | |
| 13:59:10 | sean-k-mooney | currently all jobs are queued | |
| 13:59:47 | openstackgerrit | Adam Spiers proposed openstack/nova master: Reject live migration and suspend on SEV guests https://review.opendev.org/680158 | |
| 14:00:15 | luyao | dansmith, efried: I updated the patch for 'resource' object, many thanks for your detailed comments. It helps me a lot. https://review.opendev.org/#/c/678448/17 | |
| 14:00:22 | cdent | mriedem, dansmith : I'd be inclined to think that the non-proxy original code was wrong and the fix is good and we've tended to get lucky. However, without any clear proof that presenting problem(s) is fixed by the change...hard to say. I would think merge to master and watch wouldn't be a bad thing. | |
| 14:00:29 | sean-k-mooney | i proably should have kicked off one of the later patces to run less tests | |
| 14:00:42 | sean-k-mooney | but i did want that to report back in anycase | |
| 14:00:51 | aspiers | mriedem: thanks for feedback; addressed - https://review.opendev.org/#/c/680158/4..5/nova/compute/api.py | |
| 14:01:29 | openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | |
| 14:01:56 | alex_xu | anyone can tell me...can we boot instance without allocation | |
| 14:02:22 | sean-k-mooney | am we should be able to start them. but create new instances no | |
| 14:02:50 | sean-k-mooney | e.g. you deleted the allocation in placmenet for an instance nova should still be able to start/stop the vm | |
| 14:03:26 | dansmith | cdent: specifically one customer clearly *still* has the problem after applying this patch | |
| 14:03:39 | cdent | ah | |
| 14:03:40 | cdent | well | |
| 14:03:52 | dansmith | cdent: but I agree that it *seems* like the change we'd want to make, I'd just like confirmation that it does something positive you know :) | |
| 14:04:02 | alex_xu | sean-k-mooney: probably we can before https://github.com/openstack/nova/blob/master/nova/tests/functional/test_nova_manage.py#L405 | |
| 14:04:03 | cdent | is there concrete evidence it is stuck in eventlet, or just that it is stuck? | |
| 14:05:01 | dansmith | cdent: there's some guru meditation that makes it look like it, but we don't get to control when that happens and thus it's pretty likely to always be blocking on some C call on a busy system I think | |
| 14:05:02 | artom | mriedem, FWIW, I've done an inital read-through of all your comments, and they all make sense, so I should have answers/fixes later today | |
| 14:06:16 | alex_xu | sean-k-mooney: that functional test ensure we can boot instance without allocation, but the goal is testing the heal_allocations. | |
| 14:06:40 | cdent | dansmith: terrific! Does it happen often? Therapuetic process kills to the rescue? | |
| 14:07:23 | dansmith | yeah, just nuke the compute service I think | |
| 14:07:24 | dansmith | and sometimes they can't reproduce it, like when we got on a call to watch :) | |
| 14:07:46 | dansmith | what's strange to me is that this code has been in place for ages and suddenly there are two customers hitting it simultaneously, but not fixed by this patch | |
| 14:07:53 | dansmith | just makes me highly skeptical :) | |
| 14:08:41 | cdent | yeah. eventlet, libc, libvirt changes all possible contributors | |
| 14:10:38 | melwitt | does anyone know anything about the nova.task_log table? we have customers whose nova.task_log table grows indefinitely and I found there's no methods for ever deleting task_log entries. does anyone know what people usually do about this? manually truncate the task_log table? what are these records for? is arbitrary truncation safe? | |
| 14:10:46 | mriedem | dtantsur: dustinc: efried: what do you think about these comments? https://review.opendev.org/#/c/680684/2 | |
| 14:10:54 | mriedem | the fix is fine, but wondering about the 'this shouldn't happen' case | |
| 14:13:31 | dtantsur | will check in a few minutes | |
| 14:13:42 | dtantsur | yeah, I was also unsure whether to log or to raise | |
| 14:15:05 | mriedem | alex_xu: with the caching scheduler yes it was possible to create an instance with no allocations in placement, | |
| 14:15:14 | mriedem | because the caching scheduler didn't use placement or GET /allocatoin_candidates | |
| 14:15:17 | mriedem | the caching scheduler was removed in stein, | |
| 14:15:24 | mriedem | but you can still load up out of tree scheduler drivers that may not use placement | |
| 14:15:41 | mriedem | alex_xu: coincidentally, see https://review.opendev.org/#/c/680521/ | |
| 14:16:23 | cdent | mriedem: that mention of logging reminds me of something I wanted to ask about: In python 2, when using oslo_log and LOG.error it includes whatever the most recent exception was, even if it is out of scope. In pytyhon3 sys.exc_info() "expires" sooner, more predictably. This means that error messages in the scheduler report client often include bogus tails in the message | |
| 14:16:39 | cdent | do we care? or can we just ignore python2 soon enough? | |
| 14:18:03 | mriedem | cdent: i think i know what you're talking about, and melwitt/dhellmann had made some fixes to oslo.log to squash that weirdness, but maybe that was only for warning level logs, | |
| 14:18:29 | mriedem | e.g. you'd see a stacktrace for a ComputeNodeNotFound_Remote warning even though you weren't using exc_info=True on LOG.warning, | |
| 14:18:30 | mriedem | something like that | |
| 14:18:30 | cdent | it's warning and worse where oslo_log does the inspection exc_info | |
| 14:18:56 | melwitt | oh yeah, good memory | |
| 14:19:37 | melwitt | https://review.opendev.org/528036 | |
| 14:19:50 | mriedem | melwitt: i'm pretty sure task log is related to https://docs.openstack.org/api-ref/compute/#server-usage-audit-log-os-instance-usage-audit-log which is related to ceilometer which no one uses | |
| 14:19:57 | mriedem | i'd like to burn out all of that code | |
| 14:20:35 | melwitt | thanks. I was surprised in all my time working on nova, I've never heard of the task_log table before now | |
| 14:21:04 | melwitt | so I'd support burning the code that no one uses if indeed no one uses it | |
| 14:21:08 | efried | dtantsur: I looked briefly at the ironic CI job to see why it didn't hit that bug; I think it's just because those methods are used in the course of specific operations that tempest doesn't cover. We should add that, for sure. I'm not completely sure how to get started with that though. | |
| 14:21:30 | dtantsur | yeah | |
| 14:21:39 | dtantsur | the 2nd bug is hit when you have >1 nodes | |
| 14:21:47 | mriedem | rackspace probably had internal scripts or something to purge task log | |
| 14:22:02 | mriedem | melwitt: well clearly your customer is using it | |
| 14:22:09 | cdent | thanks melwitt, I think that corresponds to something I was looking at when poking at the problem, but doesn't address the issue with the exception name being included in the log even though it was "elsewhere". I'm inclined to ignore it if nobody is complaining, since it gets better with python3. I only noticed becasue somebody asked why there was a "NotImplemted" at the end of rp generation conflict message | |