Earlier  
Posted Nick Remark
#openstack-nova - 2017-09-20
13:02:14 stephenfin sahid: RE: https://review.openstack.org/#/c/501132/, could you add a summary of the comments with sean-k-mooney to the commit message? After that, it's an easy +2
13:05:45 alex_xu nova api meeting is running at #openstack-meeting-4
13:14:15 openstackgerrit Merged openstack/nova-specs master: Spec to remove the hide server address config options https://review.openstack.org/502516
13:28:23 openstackgerrit Eric Fried proposed openstack/nova master: Use ksa adapter for placement conf & requests https://review.openstack.org/492247
13:46:48 openstackgerrit Merged openstack/nova master: Fix a typo https://review.openstack.org/505062
13:49:34 openstackgerrit Stephen Finucane proposed openstack/nova master: doc: Split flavors docs into admin and user guides https://review.openstack.org/501342
13:49:35 openstackgerrit Stephen Finucane proposed openstack/nova master: doc: Add documentation for cpu_realtime, cpu_realtime_mask https://review.openstack.org/502056
13:49:35 openstackgerrit Stephen Finucane proposed openstack/nova master: doc: Add documentation for emulator_thread_policy https://review.openstack.org/501721
13:50:10 stephenfin sdague: Could you take a look at the first of those? gibi has reviewed it a few times but it keeps getting into merge conflicts :(
13:52:49 sdague +2
13:52:55 openstackgerrit sahid proposed openstack/nova master: libvirt: bandwidth param should be set in guest migrate https://review.openstack.org/497455
13:52:56 sdague that all looks very reasonable
13:52:56 openstackgerrit sahid proposed openstack/nova master: libvirt: slow live-migration to ensure network is ready https://review.openstack.org/497457
13:52:56 openstackgerrit sahid proposed openstack/nova master: libvirt: add method to configure migration speed https://review.openstack.org/497456
13:53:19 stephenfin sdague: Thank you, sir
13:53:20 gibi stephenfin, sdague: I'm also checking that rebase with an intent to approve it
13:53:34 stephenfin gibi: and you too :)
13:55:33 stephenfin sahid: Could you take a look at https://review.openstack.org/#/c/502056/ again?
13:56:05 stephenfin I get that we need to do more configuration that what's there, but there's a lot of stuff to do. I'd prefer to add an admin guide doc for that in the future
13:56:16 stephenfin ...which I should do sometime in the next few weeks
13:57:04 mdbooth 2017-09-14 15:54:39.689 120626 ERROR nova.s2017-09-14 15:54:39.690 120562 ERROR nova.servicegroup.drivers.db [-] Unexpected error while reporting service status
13:57:16 gibi stephenfin: +2+W
13:57:20 openstackgerrit OpenStack Proposal Bot proposed openstack/os-traits master: Updated from global requirements https://review.openstack.org/503646
13:57:23 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708
13:57:25 stephenfin gibi: Yay. Thanks :)
13:57:35 mdbooth With the continuation of the first log later in the file without its timestamp
13:57:43 mdbooth Are we not locking in the logger?
13:58:22 sahid stephenfin: the admin is going to configure the flavor, it seems reasonable to me to add a note saying what i do have mentioned on the review
13:58:42 mdbooth Unfortunately, it also means the logs are not correctly sorted :/
13:58:58 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: [WIP] CLI for aggregates https://review.openstack.org/505643
13:58:58 stephenfin sahid: Right, but vcpu_pin_set is not a flavor property. I've mentioned pinned CPUs because it is
13:59:03 mdbooth This is not conducive to merge sorting
13:59:37 stephenfin You also need to configure things like isolate the CPUs and use a properly configured guest, but I don't mention those there because they're nothing to do with configuring flavor properties
13:59:55 sahid isolate the CPUs?
14:00:13 stephenfin sahid: From the host?
14:00:29 stephenfin Whatever it is that replace the isolcpus boot parameter
14:00:31 sahid that is not related to Nova, i'm talking about an option which is related to Nova, that is why i think a small note is important
14:01:25 stephenfin and vcpu_pin_set is not related to flavors. This is an flavor (extra_spec) overview doc
14:01:35 sahid if you don't mention that an admin could just think that after to have configured the host, enabling cpu_realtime=yes in flavor is enough
14:02:41 stephenfin They could also boot an standard Linux guest kernel or forget to configure isolcpus
14:03:00 stephenfin The point is that this is just an overview of the flavor extra specs available and the interactions between them
14:03:18 stephenfin We should add a real-time doc but it should be separate, like this:
14:03:55 stephenfin https://docs.openstack.org/nova/latest/admin/cpu-topologies.html
14:04:18 sahid stephenfin: ok, i just gave to you my point, i would have added that note but feel free to not mention it
14:05:07 stephenfin sahid: Yup, and I appreciate it :) I'm countering that I don't think it's necessary here, and would make more sense in the upcoming larger doc
14:05:16 stephenfin ...where I'll definitely mention it
14:06:02 mriedem mdbooth: it's likely a problem in the customers log config
14:06:13 mriedem mdbooth: i saw something like that with our new super conductor logs in devstack, the fix for that was in devstack https://review.openstack.org/#/c/497944/1/lib/nova
14:06:22 mdbooth mriedem: Looking.
14:06:22 mriedem but you should checkout what devstack does for log config
14:08:03 mdbooth mriedem: Are you sure that's the same? It looks like 2 threads are writing simultaneously to the same log file.
14:08:36 mdbooth Hence a new log starts in the middle of the previous one, rather than on a separate line
14:08:39 sdague mdbooth: the python logger should handle that
14:09:51 sdague callers of the logger should not be locking around it, that's all supposed to be handled within the logger itself
14:10:07 sdague all oslo.log does is setup some common python logger patterns
14:10:08 mdbooth sdague: Yeah, that's what I'd have thought...
14:10:33 mdbooth I wonder how this has happened, though
14:10:51 mdbooth There are a ton of examples of it in these logs
14:11:18 sdague mdbooth: going through syslog?
14:11:30 sdague because syslog has some challenges
14:11:36 mdbooth Looks like it was generating DB errors continuously for a period of time, so lots of opportunity for overlap
14:11:49 mdbooth Do we normally log through syslog?
14:11:53 toabctl mriedem, hey. could you please have another look at https://review.openstack.org/#/c/398308/ ?
14:11:58 sdague no, we normally log to a file or stdout
14:12:37 sdague now, that being said, I expect if you are blowing through the output buffer regularly by putting giant stack traces all the time, you might end up with the tails of those going weird
14:12:51 sdague but that's probably a more latent python logging issue
14:12:51 jamespage sdague, efried: just to be clear, I don't think there is a fix to make in qemu - afaict its behaving as intended for the 2.10 release
14:13:07 sdague jamespage: yeh, it being another flag seems to indicate that
14:13:19 sdague jamespage: do you all have a patch already for it for your pike ppa on nova?
14:13:20 mdbooth Hmm, this is conductor not compute
14:13:34 mdbooth Are the workers independent?
14:13:48 mdbooth i.e. might they be separately opening the same log file?
14:14:26 sdague mdbooth: yes, the workers are processes
14:14:28 jamespage sdague: I have a simple patch to fix the nova package for Pike in Ubuntu and the UCA; that's not good for direct submission to nova as its not conditional i.e. its a blind add the flag (cause we know which qemu version will be in use for artful and xenial+ Pike UCA)
14:14:37 mdbooth sdague: I'll bet that's it...
14:14:41 sdague jamespage: gotcha
14:14:48 mriedem toabctl: done
14:15:02 toabctl mriedem, thx
14:15:06 sdague jamespage: link for where you injected it might be good regardless
14:15:17 sdague jamespage: then we can figure out the right conditional
14:15:30 jamespage sdague: looking the libvirt driver + images module to figure out the best way to pass that in conditionally - most version checking is done in driver, not in images...
14:15:34 jamespage sdague:
14:15:35 jamespage sure
14:15:56 mriedem jamespage: i was having the same problem when thinking about how to make that conditional
14:16:04 mriedem the driver knows the version, but way down in the bowels of the image code it doesn't
14:16:20 jamespage mriedem: yeah its awkward from that perspective
14:16:32 jamespage lemme attach my patch to the bug report
14:18:57 mriedem tasker: updated https://review.openstack.org/#/c/504260/
14:20:09 jamespage mriedem: my thinking was to pass that down from driver to the image code as an optional param
14:20:46 jamespage testing that patch shortly
14:22:32 mriedem jamespage: yeah that is probably what i'd do
14:23:15 mriedem exit code is 1 when it fails, so that's not unique enough,
14:23:29 mriedem we could scrape the stderr for the message, and retry with the flag, but that's not fun either
14:23:49 openstackgerrit Dan Smith proposed openstack/nova master: Use improved instance_list module in compute API https://review.openstack.org/505418
14:23:50 openstackgerrit Dan Smith proposed openstack/nova master: Remove legacy fault-loading routines https://review.openstack.org/505456
14:23:52 openstackgerrit Dan Smith proposed openstack/nova master: Fix a pagination logic bug in test_bug_1689692 https://review.openstack.org/505661
14:24:09 sahid stephenfin: about https://review.openstack.org/#/c/501132/, it seems to me the commit message well reflects what is done on the patch
14:24:17 sdague mriedem: is there a reason to not set a CONST with the version on __init__ of the driver?
14:25:06 stephenfin sahid: It reflects what but not _why_. The why is what I care about (I can parse the what from reading the code)
14:25:38 mriedem sdague: and have the image code check the driver?

Earlier   Later