Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-06
13:51:24 dansmith maybe we could get efried to comment on the risk, and maybe cdent on the actual eventlet bits?
13:51:32 dansmith I'm not sure what context cdent has on eventlet, but it seems like sometimes he has a lot
13:51:44 cdent hello what?
13:51:49 mriedem efried isn't a stable branch maintainer though, or you just mean risk for FF?
13:51:49 cdent which thing?
13:52:01 dansmith mriedem: I mean for merging to master right now or in general yeah
13:52:10 mriedem cdent: https://review.opendev.org/#/c/677736/
13:52:20 mriedem cdent: tl;dr old ass code,
13:52:40 mriedem suddenly 2 rhosp customers are seeing weird issues which mdbooth thinks are related to libvirt + python + eventlet,
13:52:41 dansmith (I wish I was working on things that *have* to merge by FF right now)
13:52:52 mriedem but there isn't a clear indication that patch fixes those problems
13:52:54 openstackgerrit Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616
13:53:07 aspiers stephenfin: Your doc feedback should be addressed now ^^^
13:53:16 dansmith mriedem: I see no indication that it fixes those problems.. I think the only indication is that it "seems wrong" and "these tests pass"
13:53:36 mriedem sure, so ruling it out i guess
13:53:53 mriedem i also need to investigate these fixes that are blocking the ironic gate atm
13:54:01 mriedem since i'm guessing ironic would like to merge code before FF
13:54:15 aspiers stephenfin: https://review.opendev.org/#/c/666616/47..48/nova/conf/libvirt.py
13:54:18 donnyd sean-k-mooney: Is the NUMA job still throwing errors?
13:54:34 aspiers stephenfin: BTW I've checked that the rst all renders OK
13:54:54 sean-k-mooney donnyd: i have not re run it
13:55:08 sean-k-mooney with the queue thats in the gate i was going to wiat until the weekend
13:55:23 sean-k-mooney donnyd: i can kick it off now if you like and see
13:55:30 donnyd sure
13:58:07 sean-k-mooney donnyd: there is a build stitting in the experimental pipeline now
13:58:16 sean-k-mooney donnyd: ill let you know how it goes
13:58:20 donnyd kk
13:58:29 donnyd where can i check its status?
13:58:39 sean-k-mooney http://zuul.openstack.org/status
13:58:47 sean-k-mooney filter by 679656
13:59:10 sean-k-mooney currently all jobs are queued
13:59:47 openstackgerrit Adam Spiers proposed openstack/nova master: Reject live migration and suspend on SEV guests https://review.opendev.org/680158
14:00:15 luyao dansmith, efried: I updated the patch for 'resource' object, many thanks for your detailed comments. It helps me a lot. https://review.opendev.org/#/c/678448/17
14:00:22 cdent mriedem, dansmith : I'd be inclined to think that the non-proxy original code was wrong and the fix is good and we've tended to get lucky. However, without any clear proof that presenting problem(s) is fixed by the change...hard to say. I would think merge to master and watch wouldn't be a bad thing.
14:00:29 sean-k-mooney i proably should have kicked off one of the later patces to run less tests
14:00:42 sean-k-mooney but i did want that to report back in anycase
14:00:51 aspiers mriedem: thanks for feedback; addressed - https://review.opendev.org/#/c/680158/4..5/nova/compute/api.py
14:01:29 openstackgerrit Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616
14:01:56 alex_xu anyone can tell me...can we boot instance without allocation
14:02:22 sean-k-mooney am we should be able to start them. but create new instances no
14:02:50 sean-k-mooney e.g. you deleted the allocation in placmenet for an instance nova should still be able to start/stop the vm
14:03:26 dansmith cdent: specifically one customer clearly *still* has the problem after applying this patch
14:03:39 cdent ah
14:03:40 cdent well
14:03:52 dansmith cdent: but I agree that it *seems* like the change we'd want to make, I'd just like confirmation that it does something positive you know :)
14:04:02 alex_xu sean-k-mooney: probably we can before https://github.com/openstack/nova/blob/master/nova/tests/functional/test_nova_manage.py#L405
14:04:03 cdent is there concrete evidence it is stuck in eventlet, or just that it is stuck?
14:05:01 dansmith cdent: there's some guru meditation that makes it look like it, but we don't get to control when that happens and thus it's pretty likely to always be blocking on some C call on a busy system I think
14:05:02 artom mriedem, FWIW, I've done an inital read-through of all your comments, and they all make sense, so I should have answers/fixes later today
14:06:16 alex_xu sean-k-mooney: that functional test ensure we can boot instance without allocation, but the goal is testing the heal_allocations.
14:06:40 cdent dansmith: terrific! Does it happen often? Therapuetic process kills to the rescue?
14:07:23 dansmith yeah, just nuke the compute service I think
14:07:24 dansmith and sometimes they can't reproduce it, like when we got on a call to watch :)
14:07:46 dansmith what's strange to me is that this code has been in place for ages and suddenly there are two customers hitting it simultaneously, but not fixed by this patch
14:07:53 dansmith just makes me highly skeptical :)
14:08:41 cdent yeah. eventlet, libc, libvirt changes all possible contributors
14:10:38 melwitt does anyone know anything about the nova.task_log table? we have customers whose nova.task_log table grows indefinitely and I found there's no methods for ever deleting task_log entries. does anyone know what people usually do about this? manually truncate the task_log table? what are these records for? is arbitrary truncation safe?
14:10:46 mriedem dtantsur: dustinc: efried: what do you think about these comments? https://review.opendev.org/#/c/680684/2
14:10:54 mriedem the fix is fine, but wondering about the 'this shouldn't happen' case
14:13:31 dtantsur will check in a few minutes
14:13:42 dtantsur yeah, I was also unsure whether to log or to raise
14:15:05 mriedem alex_xu: with the caching scheduler yes it was possible to create an instance with no allocations in placement,
14:15:14 mriedem because the caching scheduler didn't use placement or GET /allocatoin_candidates
14:15:17 mriedem the caching scheduler was removed in stein,
14:15:24 mriedem but you can still load up out of tree scheduler drivers that may not use placement
14:15:41 mriedem alex_xu: coincidentally, see https://review.opendev.org/#/c/680521/
14:16:23 cdent mriedem: that mention of logging reminds me of something I wanted to ask about: In python 2, when using oslo_log and LOG.error it includes whatever the most recent exception was, even if it is out of scope. In pytyhon3 sys.exc_info() "expires" sooner, more predictably. This means that error messages in the scheduler report client often include bogus tails in the message
14:16:39 cdent do we care? or can we just ignore python2 soon enough?
14:18:03 mriedem cdent: i think i know what you're talking about, and melwitt/dhellmann had made some fixes to oslo.log to squash that weirdness, but maybe that was only for warning level logs,
14:18:29 mriedem e.g. you'd see a stacktrace for a ComputeNodeNotFound_Remote warning even though you weren't using exc_info=True on LOG.warning,
14:18:30 mriedem something like that
14:18:30 cdent it's warning and worse where oslo_log does the inspection exc_info
14:18:56 melwitt oh yeah, good memory
14:19:37 melwitt https://review.opendev.org/528036
14:19:50 mriedem melwitt: i'm pretty sure task log is related to https://docs.openstack.org/api-ref/compute/#server-usage-audit-log-os-instance-usage-audit-log which is related to ceilometer which no one uses
14:19:57 mriedem i'd like to burn out all of that code
14:20:35 melwitt thanks. I was surprised in all my time working on nova, I've never heard of the task_log table before now
14:21:04 melwitt so I'd support burning the code that no one uses if indeed no one uses it
14:21:08 efried dtantsur: I looked briefly at the ironic CI job to see why it didn't hit that bug; I think it's just because those methods are used in the course of specific operations that tempest doesn't cover. We should add that, for sure. I'm not completely sure how to get started with that though.
14:21:30 dtantsur yeah
14:21:39 dtantsur the 2nd bug is hit when you have >1 nodes
14:21:47 mriedem rackspace probably had internal scripts or something to purge task log
14:22:02 mriedem melwitt: well clearly your customer is using it
14:22:09 cdent thanks melwitt, I think that corresponds to something I was looking at when poking at the problem, but doesn't address the issue with the exception name being included in the log even though it was "elsewhere". I'm inclined to ignore it if nobody is complaining, since it gets better with python3. I only noticed becasue somebody asked why there was a "NotImplemted" at the end of rp generation conflict message
14:22:46 melwitt mriedem: oh, you think table grow means they are using the audit APIs? I had assumed task_log entries get added regardless (like on any instance action)
14:22:49 dtantsur mriedem: better ^^?
14:22:49 openstackgerrit Dmitry Tantsur proposed openstack/nova master: Fix incorrect invocation of openstacksdk's baremetal.nodes() https://review.opendev.org/680684
14:23:15 alex_xu mriedem: I'm talking to luyao, we should stop the instance booting if there is no allocation, since vgpu and vpmem should depend on allocation
14:23:22 melwitt I need to check
14:23:25 mriedem melwitt: it's a periodic task in the compute service which you have to enable, and it's disabled by default
14:23:32 mriedem so if they have task log table entries they are configured to run that task
14:23:34 melwitt oh, dang
14:23:42 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.instance_usage_audit
14:23:56 mriedem and they are probably using ceilometer to hit that compute api
14:24:03 melwitt thanks. I shouldn't have assumed
14:24:03 mriedem if they aren't, then they should disable the task and purge the db
14:24:13 mriedem please see topic new contributor :P
14:24:24 melwitt :(
14:25:06 mriedem alex_xu: yeah that's probably fine, there is no real contract on support for out of tree scheduler drivers
14:25:34 mriedem alex_xu: see https://opendev.org/openstack/nova/src/branch/master/nova/scheduler/driver.py#L35

Earlier   Later