Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-06
13:58:20 donnyd kk
13:58:29 donnyd where can i check its status?
13:58:39 sean-k-mooney http://zuul.openstack.org/status
13:58:47 sean-k-mooney filter by 679656
13:59:10 sean-k-mooney currently all jobs are queued
13:59:47 openstackgerrit Adam Spiers proposed openstack/nova master: Reject live migration and suspend on SEV guests https://review.opendev.org/680158
14:00:15 luyao dansmith, efried: I updated the patch for 'resource' object, many thanks for your detailed comments. It helps me a lot. https://review.opendev.org/#/c/678448/17
14:00:22 cdent mriedem, dansmith : I'd be inclined to think that the non-proxy original code was wrong and the fix is good and we've tended to get lucky. However, without any clear proof that presenting problem(s) is fixed by the change...hard to say. I would think merge to master and watch wouldn't be a bad thing.
14:00:29 sean-k-mooney i proably should have kicked off one of the later patces to run less tests
14:00:42 sean-k-mooney but i did want that to report back in anycase
14:00:51 aspiers mriedem: thanks for feedback; addressed - https://review.opendev.org/#/c/680158/4..5/nova/compute/api.py
14:01:29 openstackgerrit Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616
14:01:56 alex_xu anyone can tell me...can we boot instance without allocation
14:02:22 sean-k-mooney am we should be able to start them. but create new instances no
14:02:50 sean-k-mooney e.g. you deleted the allocation in placmenet for an instance nova should still be able to start/stop the vm
14:03:26 dansmith cdent: specifically one customer clearly *still* has the problem after applying this patch
14:03:39 cdent ah
14:03:40 cdent well
14:03:52 dansmith cdent: but I agree that it *seems* like the change we'd want to make, I'd just like confirmation that it does something positive you know :)
14:04:02 alex_xu sean-k-mooney: probably we can before https://github.com/openstack/nova/blob/master/nova/tests/functional/test_nova_manage.py#L405
14:04:03 cdent is there concrete evidence it is stuck in eventlet, or just that it is stuck?
14:05:01 dansmith cdent: there's some guru meditation that makes it look like it, but we don't get to control when that happens and thus it's pretty likely to always be blocking on some C call on a busy system I think
14:05:02 artom mriedem, FWIW, I've done an inital read-through of all your comments, and they all make sense, so I should have answers/fixes later today
14:06:16 alex_xu sean-k-mooney: that functional test ensure we can boot instance without allocation, but the goal is testing the heal_allocations.
14:06:40 cdent dansmith: terrific! Does it happen often? Therapuetic process kills to the rescue?
14:07:23 dansmith yeah, just nuke the compute service I think
14:07:24 dansmith and sometimes they can't reproduce it, like when we got on a call to watch :)
14:07:46 dansmith what's strange to me is that this code has been in place for ages and suddenly there are two customers hitting it simultaneously, but not fixed by this patch
14:07:53 dansmith just makes me highly skeptical :)
14:08:41 cdent yeah. eventlet, libc, libvirt changes all possible contributors
14:10:38 melwitt does anyone know anything about the nova.task_log table? we have customers whose nova.task_log table grows indefinitely and I found there's no methods for ever deleting task_log entries. does anyone know what people usually do about this? manually truncate the task_log table? what are these records for? is arbitrary truncation safe?
14:10:46 mriedem dtantsur: dustinc: efried: what do you think about these comments? https://review.opendev.org/#/c/680684/2
14:10:54 mriedem the fix is fine, but wondering about the 'this shouldn't happen' case
14:13:31 dtantsur will check in a few minutes
14:13:42 dtantsur yeah, I was also unsure whether to log or to raise
14:15:05 mriedem alex_xu: with the caching scheduler yes it was possible to create an instance with no allocations in placement,
14:15:14 mriedem because the caching scheduler didn't use placement or GET /allocatoin_candidates
14:15:17 mriedem the caching scheduler was removed in stein,
14:15:24 mriedem but you can still load up out of tree scheduler drivers that may not use placement
14:15:41 mriedem alex_xu: coincidentally, see https://review.opendev.org/#/c/680521/
14:16:23 cdent mriedem: that mention of logging reminds me of something I wanted to ask about: In python 2, when using oslo_log and LOG.error it includes whatever the most recent exception was, even if it is out of scope. In pytyhon3 sys.exc_info() "expires" sooner, more predictably. This means that error messages in the scheduler report client often include bogus tails in the message
14:16:39 cdent do we care? or can we just ignore python2 soon enough?
14:18:03 mriedem cdent: i think i know what you're talking about, and melwitt/dhellmann had made some fixes to oslo.log to squash that weirdness, but maybe that was only for warning level logs,
14:18:29 mriedem e.g. you'd see a stacktrace for a ComputeNodeNotFound_Remote warning even though you weren't using exc_info=True on LOG.warning,
14:18:30 mriedem something like that
14:18:30 cdent it's warning and worse where oslo_log does the inspection exc_info
14:18:56 melwitt oh yeah, good memory
14:19:37 melwitt https://review.opendev.org/528036
14:19:50 mriedem melwitt: i'm pretty sure task log is related to https://docs.openstack.org/api-ref/compute/#server-usage-audit-log-os-instance-usage-audit-log which is related to ceilometer which no one uses
14:19:57 mriedem i'd like to burn out all of that code
14:20:35 melwitt thanks. I was surprised in all my time working on nova, I've never heard of the task_log table before now
14:21:04 melwitt so I'd support burning the code that no one uses if indeed no one uses it
14:21:08 efried dtantsur: I looked briefly at the ironic CI job to see why it didn't hit that bug; I think it's just because those methods are used in the course of specific operations that tempest doesn't cover. We should add that, for sure. I'm not completely sure how to get started with that though.
14:21:30 dtantsur yeah
14:21:39 dtantsur the 2nd bug is hit when you have >1 nodes
14:21:47 mriedem rackspace probably had internal scripts or something to purge task log
14:22:02 mriedem melwitt: well clearly your customer is using it
14:22:09 cdent thanks melwitt, I think that corresponds to something I was looking at when poking at the problem, but doesn't address the issue with the exception name being included in the log even though it was "elsewhere". I'm inclined to ignore it if nobody is complaining, since it gets better with python3. I only noticed becasue somebody asked why there was a "NotImplemted" at the end of rp generation conflict message
14:22:46 melwitt mriedem: oh, you think table grow means they are using the audit APIs? I had assumed task_log entries get added regardless (like on any instance action)
14:22:49 dtantsur mriedem: better ^^?
14:22:49 openstackgerrit Dmitry Tantsur proposed openstack/nova master: Fix incorrect invocation of openstacksdk's baremetal.nodes() https://review.opendev.org/680684
14:23:15 alex_xu mriedem: I'm talking to luyao, we should stop the instance booting if there is no allocation, since vgpu and vpmem should depend on allocation
14:23:22 melwitt I need to check
14:23:25 mriedem melwitt: it's a periodic task in the compute service which you have to enable, and it's disabled by default
14:23:32 mriedem so if they have task log table entries they are configured to run that task
14:23:34 melwitt oh, dang
14:23:42 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.instance_usage_audit
14:23:56 mriedem and they are probably using ceilometer to hit that compute api
14:24:03 melwitt thanks. I shouldn't have assumed
14:24:03 mriedem if they aren't, then they should disable the task and purge the db
14:24:13 mriedem please see topic new contributor :P
14:24:24 melwitt :(
14:25:06 mriedem alex_xu: yeah that's probably fine, there is no real contract on support for out of tree scheduler drivers
14:25:34 mriedem alex_xu: see https://opendev.org/openstack/nova/src/branch/master/nova/scheduler/driver.py#L35
14:27:41 efried dansmith: risk on https://review.opendev.org/#/c/677736/ I really have no idea I'm afraid. If cdent thinks it's okay to merge-and-watch, that's good enough for me. Remember FF is FF so we have a little bit of time before the release to flush out issues; so I don't think current proximity to FF is a reason to block it.
14:28:33 dansmith efried: ack
14:33:36 mriedem dansmith: before artom moves forward with this rpc/object change in the numa lm series https://review.opendev.org/#/c/634827/57//COMMIT_MSG@12 might want to see if you are on board
14:33:56 mriedem i'm basically asking that we're consistent in the flags saying both source and dest are ready for action
14:33:57 alex_xu mriedem: i see now, thanks
14:34:05 dansmith mriedem: ack looking
14:34:15 artom mriedem, actually yeah, thanks. I think it should be fine, but dansmith's lack of veto would boost confidence
14:35:16 dansmith artom: mriedem you're thinking set that field on the dest using the same can-do-numa thing in rpcapi? because without that, the only way to know is to pass a new param and have it *not* show up at the other end
14:35:34 dansmith we resolved that first, and then artom added the can-do check to rpcapi for the source, who has nobody to call
14:35:48 dansmith so if you're saying make the source and dest both call that, and set their corresponding field, yeah I guess that's okay
14:37:22 artom dansmith, not quite - I'm saying the dest doesn't need to actually send RPC 5.3, so check_can_live_migrate_source doesn't need that numa_live_migraiton flag, the dest can just shove it in migrate_data
14:38:17 aspiers Just asked in #openstack-qa but figured worth asking here too: Is it considered good/bad practice for tempest tests to check the message string of nova exceptions returned by assertRaises?
14:38:37 dansmith artom: but the dest needs to support the 5.3 inbound call, but you're saying actually report the conductor side? meaning the did-i-get-the-migration-from-conductor flag?
14:38:59 aspiers I'm thinking that if we wanted to reword an exception's message string, this could break tempest, and then you'd need a circular Depends-On across repos to fix it
14:39:10 artom dansmith, right, the dest needs to receive 5.3 - specifically migration and limits form the conductor
14:39:20 artom So if gets those, it can set the flag inside migrate_data
14:39:24 dansmith artom: and the new drop call
14:39:27 dansmith from source
14:39:31 artom dansmith, yeah
14:39:35 artom No, not from source
14:39:40 artom The source still needs to send 5.3
14:39:43 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix incorrect invocation of openstacksdk's baremetal.nodes() https://review.opendev.org/680684
14:39:51 mriedem dtantsur: +2 now ^
14:39:52 mriedem efried: ^
14:39:53 dansmith artom: the source calls the new drop-claim against the dest right?
14:40:02 artom dansmith, yeah

Earlier   Later