Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-15
13:34:49 coreycb mriedem: i wonder if they could work around it by running with python3
13:35:19 kashyap dansmith: Yeah, I agre with the former (not put them in traits). The latter is a 'policy' decision -- so I'm fine with whatever the consensus of the community is.
13:39:37 amorin hey all, can someone take a look at that: https://review.opendev.org/#/c/658362/
13:45:28 mdbooth coreycb: Are you talking about the eventlet thing?
13:45:51 coreycb mdbooth: ye
13:45:52 coreycb yes
13:46:05 mdbooth coreycb: Unfortunately not. We're hitting this in OSP15, which is py3 only.
13:46:26 sean-k-mooney coreycb: if you dont use uswign and run the nova-api via the console script it shoudl not have this issue
13:46:39 mdbooth However, IIRC there's another project also hitting it?
13:46:39 sean-k-mooney *uwsgi
13:46:41 coreycb mdbooth: ok. i just saw py2.7 in the traceback and that was my thought. sean-k-mooney: ok
13:46:51 mdbooth sean-k-mooney: Do you happen to remember which one it was?
13:47:03 sean-k-mooney monasca i think
13:47:26 mdbooth It's a library thing, anyway. It's nothing specific to what Nova's doing.
13:47:34 mdbooth It's just that eventlet + wsgi is bad.
13:49:55 sean-k-mooney as i said the workaound currently is to not run the nova api under wsgi and fall back to using the python nova-api command and fronting that with a load blancer like ha proxy
13:51:39 mdbooth I mentioned in the review that I don't think the lack of ability to cancel a thread is a showstopper in practice, and certainly not as big an issue as eventlet + wsgi.
13:51:48 sean-k-mooney ah it was mistral
13:51:49 kashyap sean-k-mooney: What does "monasca" even do? Okay, let me go look up
13:52:02 mdbooth sean-k-mooney: That sounds better.
13:52:05 sean-k-mooney monsaca is a montioring service
13:52:13 kashyap Ah, thx.
13:52:28 mriedem amorin: you should have a test
13:52:28 sean-k-mooney but i just rememebre it began with m just fount th lin in the mailing list post
13:53:31 sean-k-mooney but ya long story short as mdbooth said eventlet+wsgi is just a bad idea in general
13:53:50 amorin mriedem: ok
14:12:46 efried alex_xu: looking...
14:21:44 tssurya mriedem: oh cool thansk :)
14:23:53 aarents mdbooth: thanks for reviewing https://review.opendev.org/#/c/659054/, you catch the test logic, I will add a comment in test as suggested
14:31:40 openstackgerrit Stephen Finucane proposed openstack/nova-specs master: Follow up for I9200c2b18369da8b9194a8ccbbf6f1baf6a9613c https://review.opendev.org/659300
14:31:57 stephenfin bauzas: Time to merge https://review.opendev.org/#/c/555081/ ?
14:32:24 stephenfin bauzas: I've posted a follow-up to address some of the outstanding comments and issues
14:32:35 bauzas cool then, I'll review those :)
14:32:48 bauzas but yeah I feel we agreed on saying cool cool at the PTG
14:41:44 efried alex_xu: How does that play with https://review.opendev.org/#/c/645520/ ?
15:10:39 openstackgerrit Alexandre arents proposed openstack/nova master: Fix live-migration when glance image deleted https://review.opendev.org/659054
15:22:02 openstackgerrit Arnaud Morin proposed openstack/nova master: Always Set dhcp_server in network_info https://review.opendev.org/658362
15:24:52 openstackgerrit Merged openstack/nova-specs master: Standardize CPU resource tracking https://review.opendev.org/555081
15:30:47 openstackgerrit Matthew Booth proposed openstack/nova stable/stein: Fix retry of instance_update_and_get_original https://review.opendev.org/659317
15:31:36 openstackgerrit Matthew Booth proposed openstack/nova stable/rocky: Fix retry of instance_update_and_get_original https://review.opendev.org/659318
15:33:02 openstackgerrit Matthew Booth proposed openstack/nova stable/queens: Fix retry of instance_update_and_get_original https://review.opendev.org/659320
15:33:20 openstackgerrit Arnaud Morin proposed openstack/nova master: Always Set dhcp_server in network_info https://review.opendev.org/658362
15:37:23 amorin hey mriedem and all
15:37:29 amorin I added some unit tests there:
15:37:31 amorin https://review.opendev.org/#/c/658362
15:50:38 mriedem ack it's in the tab queue
16:29:24 melwitt mriedem: yes, should be ready. I just looked through the latest comments again and replied. and it looks like it hasn't fallen into merge conflict yet
16:33:06 mriedem ok
16:33:21 mriedem into the runway slot vat of acid it will be thrown
16:34:41 melwitt haha, thanks
16:39:45 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Delete allocations even if _confirm_resize raises (part 2) https://review.opendev.org/659338
16:39:45 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Add functional confirm_migration_error test https://review.opendev.org/658834
16:39:46 mriedem ganso: ^
16:44:40 ganso mriedem: thanks! will rebase on top
16:45:09 ganso mriedem: oh you already rebased
16:45:11 ganso mriedem: thanks!
16:45:30 imacdonn mriedem: pls add https://review.opendev.org/#/c/653065/ to your tab queue for revisit? I have a feeling that the issue with the functional tests has something to do with sqlalchemy sessions, or something, but may be off on the wrong track
16:45:47 imacdonn (if there's a sqlalchemy guru available for a consult, that might work too)
16:45:56 weshay kashyap ping.. looking to add the qemu-agent rpm to nova, I think that would be on a container and not bm.. not sure also wanted to verify w/ you which container that rpm should be installed on
16:46:23 weshay kashyap context is: nova.exception.QemuGuestAgentNotEnabled: QEMU guest agent is not enabled
16:46:31 weshay https://bugs.launchpad.net/tripleo/+bug/1824315
16:46:32 openstack Launchpad bug 1824315 in tripleo "periodic fedora28 standalone job failing at test_volume_boot_pattern" [Critical,In progress] - Assigned to Quique Llorente (quiquell)
16:47:21 weshay melwitt maybe you would know? ^
16:50:07 melwitt weshay: the qemu guest agent is something that would have to be installed on the guest, which is currently not done as part of any CI that I'm aware of
16:50:40 weshay melwitt ok.. some folks were saying the container but the bm makes more sense to me
16:50:54 melwitt that is, if you create a nova instance, you'd then have to install the qemu guest agent on that guest after it comes up
16:52:06 melwitt but I'm wondering why it should be needed
16:52:42 dansmith if they're trying to do something that requires it, like quiesce disks (I think)
16:53:04 melwitt weshay: that error message is just saying it's going to skip quiescing the instance bc the guest agent isn't present, but it shouldn't fail the job because of that I don't think
16:53:34 weshay melwitt it does not fail consistently, maybe 1/10 times
16:53:43 weshay maybe this is a red herring
16:54:08 dansmith if it's failing on the guest agent really not being there 1/10 times, I'd suspect the guest isn't booting fully 1/10 times or something like that
16:54:24 melwitt weshay: ok. I don't think this is the root cause of the failure because AFAICT it's just skipping that. it should say that every time I think
16:55:09 melwitt dansmith: I don't think any CI installs the guest agent, at least I was not aware of any
16:55:26 openstackgerrit Merged openstack/python-novaclient master: Cap sphinx for py2 and drop keyring dependency https://review.opendev.org/659202
16:55:26 dansmith melwitt: if they are using a fedora guest...
16:55:44 melwitt dansmith: fedora guest has it by default?
16:56:01 weshay dansmith ya.. we this in fedora deployments.. but w/ cirros guest
16:56:07 dansmith melwitt: no idea, I'm just saying, different from a cirros guest
16:56:09 dansmith okay
16:56:26 dansmith the bug shows a big stack trace.. melwitt you think we're logging that trace just while checking the skip?
16:56:50 melwitt oh. yeah, anecdotally I've not had the qemu guest agent come with any cloud image, like centos for example I've used
16:56:59 melwitt dansmith: I think so, opening up the code to confirm
16:57:31 dansmith okay we should not be logging a trace for a check thing, amirite?
16:58:04 melwitt should not, no. I see a LOG.info for it but no re-raise, so not yet sure what's going on
16:58:26 dansmith that's why I'm wondering if we're actually leaking the exception out and not just skipping
16:58:58 melwitt yeah, let's see, the trace is happening in the driver, so that must be where
16:59:08 melwitt there's no re-raise in compute/api, I meant
17:06:41 cdent efried: is ocata nova still an active concern? there's a logging buglet in the reasource tracker + report client that can be pretty misleading but the issue goes away with pike (or maybe queens)
17:09:19 cdent (goes away with pike, just checked)
17:16:29 melwitt weshay: on your latest comment on LP, the log shows the code continuing on to do the volume snapshot http://logs.openstack.org/84/655684/2/check/tripleo-ci-fedora-28-standalone/eda3002/logs/undercloud/var/log/containers/nova/nova-api.log.txt.gz#_2019-05-13_16_03_50_120 so I think that trace is unrelated
17:17:03 melwitt weshay: I don't yet find how that trace is getting logged (it shouldn't be logged), so I'm still looking for how that's happening
17:17:34 weshay k k.. thanks.. so melwitt we're saying it's a red herring and the problem is elsewhere right?
17:17:43 weshay confirming that I guess
17:18:47 melwitt weshay: right. the logging of the trace is bad and I'll have to find how that's occurring to remove it, but AFAICS the code is continuing on after skipping the quiesce and shouldn't affect the pass/fail of the CI
17:19:23 melwitt "Creating snapshot from volume"
17:19:59 weshay k
17:20:04 weshay arxcruz ^
17:20:15 melwitt I'll look at the rest of those logs to see if I can tell where it's failing
17:22:20 melwitt weshay: looking at the test results of that run, the test_volume_boot_pattern passed but test_network_basic_ops failed due to a ssh timeout http://logs.openstack.org/84/655684/2/check/tripleo-ci-fedora-28-standalone/eda3002/logs/tempest.html.gz

Earlier   Later