Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-08-03
15:00:35 mriedem so i'm trying to figure out a way to get the qemu instance logs into the n-cpu logs for debug
15:00:39 mdbooth mriedem: I think we should rewrite that polling block to consume events instead. It's also less buggy.
15:00:45 cfriesen mriedem: I was just thinking that we had all the info needed to get the file, so didn't see what the problem was....but it's not "can we get the file", but "can we determine there was a failure so we know to go get the file"
15:00:50 mdbooth As in, it was designed for this in the first place.
15:01:27 mriedem cfriesen: maybe, i don't know how configurable that path is
15:01:53 mriedem seems pretty hacky though, i'd think you could get qemu guest logs from libvirt apis
15:02:12 mdbooth mriedem: I don't think so, btw.
15:02:44 cfriesen mriedem: ah, right, we don't control all the clouds this runs on. I think it is configurable where those logs go.
15:03:05 mriedem right
15:03:13 mriedem that's why i'd need an api
15:04:10 mdbooth Basically we should switch to using libvirt events api. Extensive documentation here: http://libvirt.org/docs/libvirt-appdev-guide/en-US/html/Application_Development_Guide-Guest_Domains-Event_Not.html
15:04:33 cfriesen with a big TBD on that page?
15:04:35 mriedem mdbooth: yeah i suppose virConnectDomainEventJobCompletedCallback
15:04:50 mdbooth cfriesen: You need more? Pshaw
15:05:52 mdbooth cfriesen: It's a small TBD, anyway. Classier that way.
15:07:19 mdbooth mriedem: I wonder if we could register a libvirt error handler, and dump errors into nova compute logs as a matter of course:http://libvirt.org/docs/libvirt-appdev-guide-python/en-US/html/libvirt_application_development_guide_using_python-Error_Handling-Registering_Error_Handler.html
15:07:31 mdbooth That might achieve what you want in practise.
15:11:14 mriedem where is the error array defined?
15:11:33 mdbooth mriedem: rtfs
15:11:47 cfriesen mriedem: mdbooth: is there a libvirt bug here? I mean the source is running _live_migration_monitor() and calling guest.get_job_info(). shouldn't libvirt detect a failure?
15:11:48 mriedem ha, we already register an error handler
15:11:49 mriedem pass
15:11:49 mriedem # Just ignore instead of default outputting to stderr.
15:11:49 mriedem def _libvirt_error_handler(context, err):
15:12:02 mdbooth mriedem: hehe
15:12:07 cfriesen and if it doesn't, are we going to get an error in the callback?
15:12:20 mdbooth cfriesen: No
15:12:30 mriedem "with error being a list of information about the error being raised. "
15:12:59 mriedem i suppose it's similar to a libvirtError
15:13:39 mdbooth cfriesen: I don't recall the detail now, but at the time kashyap and I went over the libvirt and libvirt python binding code very carefully
15:13:52 mdbooth cfriesen: We're extracting everything from it which can be extracted
15:14:06 mriedem ah yup it's just the libvirtError.err list
15:14:09 mdbooth Hence my big comment explaining what we're not getting
15:15:05 mdbooth It's not designed to be used this way
15:15:38 mdbooth The intention was that you'd consume events instead. That api has been stable for much longer.
15:17:14 sdague mriedem: you'd need another grok parser
15:19:51 cfriesen mdbooth: okay, I think I got it. On another note, currently with block live migration if the guest is dirtying disk quickly the current logs don't show information about the initial block transfer.
15:20:15 mriedem would be nice if the libvirtError python binding class just had a nice __repr__
15:20:22 mriedem maybe it does already...
15:21:48 openstackgerrit Dan Smith proposed openstack/nova master: remove provider allocs in confirm/revert resize https://review.openstack.org/488510
15:21:49 openstackgerrit Dan Smith proposed openstack/nova master: Add resource utilities to scheduler utils https://review.openstack.org/490514
15:21:49 openstackgerrit Dan Smith proposed openstack/nova master: Sum allocations in the scheduler when resizing to the same host https://review.openstack.org/490085
15:22:13 dansmith jaypipes: I gotta start getting ready for my call, so I'm pushing.. the last patch is the only one that needs attention, AFAIK, a few fails in unit tests at least
15:22:31 jaypipes dansmith: got it. will take the ball.
15:24:08 dansmith jaypipes: note the new patch in the middle that adds a couple of utils and generalizes something in mriedem's patch
15:24:38 jaypipes dansmith: noted
15:25:16 jaypipes dansmith: I presume that's not the patch with test failures, though, yes? the top is the one with failures?
15:25:28 dansmith jaypipes: just your last confirm/revert one yeah
15:25:34 jaypipes got it thx
15:31:09 mriedem comments in the utils patch
15:58:33 mriedem cdent: where would one find the current placement api-ref?
15:58:42 mriedem or is that still just in builds that touch it?
15:59:15 cdent mriedem: there’s no publishing job yet as we were waiting for the stack to merge so that we could then move it to the new location and _then_ publish it
16:01:39 ildikov mriedem: hey
16:01:52 ildikov mriedem: coming to the meeting?
16:03:28 openstackgerrit Stephen Finucane proposed openstack/nova master: doc: Start using oslo_config.sphinxext https://review.openstack.org/482961
16:03:29 openstackgerrit Stephen Finucane proposed openstack/nova master: doc: Start using oslo_policy.sphinxext https://review.openstack.org/479358
16:03:29 openstackgerrit Stephen Finucane proposed openstack/nova master: doc: Rework README to reflect new doc URLs https://review.openstack.org/480074
16:03:30 openstackgerrit Stephen Finucane proposed openstack/nova master: doc: Rework index page per new sections https://review.openstack.org/478485
16:03:30 openstackgerrit Stephen Finucane proposed openstack/nova master: policies: Fix Sphinx issues https://review.openstack.org/480516
16:03:31 openstackgerrit Stephen Finucane proposed openstack/nova master: doc: Remove dead files https://review.openstack.org/478470
16:04:12 mriedem cdent: ok i was going to see if we needed any changes to the api-ref for the allocations minItems:1 thing
16:05:15 cdent mriedem: i think this one is allocations: https://review.openstack.org/#/c/470933/
16:05:17 cdent so not merged yet
16:08:48 cdent mriedem: when you use the term “latent issue” what does that actually mean?
16:09:35 mriedem not introduced in pike
16:09:37 mriedem not a regression
16:10:02 cdent thanks
16:15:06 mriedem cdent: ok questions in https://review.openstack.org/#/c/470933/
16:16:18 cdent mriedem: cool, I’ll hope andrey can look at those soon. If not, I can, I’m in the api-wg meeting now and then after that am gone (officially) until monday
16:31:11 mriedem dtantsur: jlvillal: did ironic ever go ahead with raising minimum required microversions?
16:31:51 dtantsur mriedem: nope, we haven't got to it
16:32:04 dtantsur I think we have an api-wg guideline for that, resulting from our discussions
16:33:17 mriedem ok
16:33:24 mriedem edleafe: i think the unit test failures in https://review.openstack.org/#/c/487925/ are probably related
16:33:42 mriedem it's super rare to have a random unit test failure with the ironic stuff in nova, and it's suspect when you're touching that driver
16:35:00 mriedem need some cores to look at this https://review.openstack.org/#/c/489763/ - it's a regression in pike, pretty simple fix
16:35:56 edleafe mriedem: those failures do seem to be more or less random. They don't happen locally
16:37:56 mriedem edleafe: i've never seen either of those happen in our ci though
16:39:13 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%20Expected%20call%3A%20call('node.list'%2C%20associated%3DTrue%2C%20limit%3D0)%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d
16:39:28 mriedem that only shows up in that change
16:40:22 openstackgerrit Merged openstack/nova master: reflow rpc doc to 80 columns https://review.openstack.org/490455
16:41:10 openstackgerrit Merged openstack/nova master: doc: Make use of definition lists, literals https://review.openstack.org/490481
16:44:22 mriedem bauzas: any idea why the ServerGroupAntiAffinityFilter would stop working if you aren't tracking instance changes between the compute and scheduler service? Like i know there is a race possibility, but is that filter completely dependent on those? shouldn't it fallback to check the db or something if it's not tracking instance updates?
16:44:46 mriedem because with superconductor we don't have the upcalls from the computes to the scheduler for tracking instance chnages
16:44:48 mriedem *changes
16:48:39 dansmith jaypipes: I have emerged from my hole
16:50:43 dansmith jaypipes: any progress?
16:51:12 jaypipes dansmith: finishing up test runs now
16:51:50 jaypipes dansmith: had to go into your sched utils patch and add support for boot-from-volume... :(
16:54:05 dansmith my patch didn't regress that, right?
16:54:25 dansmith I dunno what change is needed for that, but I'll check it out when you push
16:55:54 jaypipes dansmith: mriedem had pointed out that we were not handling bfv in the "cheating" section.
16:56:06 jaypipes dansmith: thus my needing to put it into the resources_from_flavor() method
16:56:22 dansmith I'm saying I don't know what needs doing is all
16:56:31 openstackgerrit Jay Pipes proposed openstack/nova master: remove provider allocs in confirm/revert resize https://review.openstack.org/488510
16:56:32 openstackgerrit Jay Pipes proposed openstack/nova master: Add resource utilities to scheduler utils https://review.openstack.org/490514
16:56:43 jaypipes dansmith: gotcha. it required passing the instance as well as flavor. pls see above.
16:56:57 jaypipes dansmith: I'm still working on the top patch (confirm resize one)
16:57:06 jaypipes dansmith: but figured I'd push to show you what I changed.

Earlier   Later