| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-24 | |||
| 13:38:55 | mriedem | sdague: so i'd say we keep them separate and actually deprecate the nova wsgi_log_format option since it's in oslo.service, and doesn't even really seem useful anymore | |
| 13:39:10 | kashyap | mdbooth: Whoops, I think I see the cause: "The issue is caused by the fact that dist type is changed from 'block' to 'file' after volume migration." | |
| 13:39:20 | bauzas | mdbooth: you'll just miss unlimited beer credits | |
| 13:39:22 | kashyap | It should _retain_ 'block' as the disk type post-migration, too. | |
| 13:39:27 | sdague | mriedem: ah, right, because we're going through the python logger the default logger in eventlet is also putting in timestamping | |
| 13:39:44 | bauzas | kashyap: that is the root cause as per the reporter indeed | |
| 13:40:07 | sdague | mriedem: ok, same patch for deprecation, or follow on? | |
| 13:40:12 | bauzas | kashyap: I lean to mark the bug as confirmed, but finding a duplicate or even a bugfix would be awesomeness | |
| 13:40:13 | mriedem | follow on | |
| 13:40:31 | sdague | mriedem: then you want to hit the +A on the current one? | |
| 13:40:41 | kashyap | bauzas: Yeah, have to look for the dupliate. First, trying to wrap my head around another unit test fix. Want to get this out before I'm out for 2-ish weeks starting 01 Aug | |
| 13:40:43 | sdague | it has to +2s, but was waiting for you to come back around | |
| 13:41:00 | mriedem | sdague: i'm goign to push a devstack change that makes nova-api run under eventlet again and depends-on this change to make sure the eventlet stuff is as expected | |
| 13:41:05 | bauzas | kashyap: np, will try to wake up my gerrit search skills | |
| 13:41:06 | kashyap | bauzas: I don't think there's a duplicate. I'm looking at this change for the first time | |
| 13:41:10 | kashyap | s/change/bug/ | |
| 13:41:22 | sdague | mriedem: sure, though the unit tests mostly cover that as well | |
| 13:41:32 | kashyap | bauzas: And the above bug you point to seems caused by this (merged) change: https://review.openstack.org/#/c/465205/ | |
| 13:41:51 | gibi | mriedem: hi! I made some progress on the missing update on the updated_at field. However fix I'm currently proposing might not what we want at the end https://review.openstack.org/#/c/486561/ | |
| 13:42:06 | bauzas | kashyap: was just digging in https://review.openstack.org/#/q/project:openstack/nova+file:%255Enova/virt/libvirt/volume.py | |
| 13:42:46 | gibi | mriedem: I will try to dig deeper in oslo.db as time allows | |
| 13:43:15 | bauzas | kashyap: ok, I'll set the bug as confirmed | |
| 13:43:36 | mriedem | gibi: ok | |
| 13:44:49 | bauzas | mriedem: would it be reasonable to target bug reports as pike-rc-potential if anyone didn't provided a fix yet (ie. not in Progress) ? | |
| 13:45:50 | bauzas | mriedem: having in-progress bugs in the bucket is cool for reviews, but I'd also like to make sure we're like telling to the world that the release is worth getting those unassigned bugs fixed so someone could step up ? | |
| 13:46:07 | mriedem | bauzas: yes | |
| 13:46:31 | bauzas | mriedem: tbc, I'm worried of any bad press of https://bugs.launchpad.net/nova/+bug/1705700 if not fixed by Ocata timeframe | |
| 13:46:32 | openstack | Launchpad bug 1705700 in OpenStack Compute (nova) "live migration does not work after volume migration" [High,Confirmed] | |
| 13:47:04 | claudiub | bauzas: we've only observed that bug on hyper-v. libvirt seems to unplug the vifs on confirm_migration. | |
| 13:48:14 | mriedem | bauzas: is it latent or a regression in pike? | |
| 13:48:27 | bauzas | mriedem: it's Ocata | |
| 13:48:47 | mriedem | so a regression introduced in ocata | |
| 13:48:50 | bauzas | mriedem: but looks like https://review.openstack.org/#/c/465205/ was backported to Ocata | |
| 13:49:17 | bauzas | mriedem: I can try to investigate further | |
| 13:49:35 | bauzas | at least the bugfix above was backported in a point release | |
| 13:49:47 | bauzas | we can ask to test some compute with some older release | |
| 13:50:05 | openstackgerrit | sahid proposed openstack/nova-specs master: virt: provide a mask to selection pCPUs to run emu. threads https://review.openstack.org/486617 | |
| 13:50:34 | edleafe | Scheduler subteam meeting in 10 minutes in #openstack-meeting-alt | |
| 13:51:51 | openstackgerrit | Jianghua Wang proposed openstack/nova-specs master: Support virtual GPU resources https://review.openstack.org/450122 | |
| 14:01:30 | edleafe | Scheduler subteam meeting running now in #openstack-meeting-alt | |
| 14:01:48 | openstackgerrit | Sean Dague proposed openstack/nova master: deprecate ``wsgi_log_format`` config variable https://review.openstack.org/486623 | |
| 14:08:42 | artom | kashyap, mdbooth, you mean | |
| 14:09:18 | artom | Err | |
| 14:09:25 | artom | kashyap, mdbooth, you mean https://review.openstack.org/#/c/471356/ ? | |
| 14:09:26 | openstackgerrit | Jay Pipes proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566 | |
| 14:09:34 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Post-migration, set cache value for Cinder volume(s) https://review.openstack.org/485752 | |
| 14:09:46 | artom | This is the yet unmerged newton backport, the other ones have merged | |
| 14:09:47 | mdbooth | bauzas: artom gets all your beer :) | |
| 14:10:50 | kashyap | artom: Ah, nice one. Yep, that's it | |
| 14:11:32 | bauzas | mdbooth: FWIW I just marked the bug we discussed as incomplete since I'd like the reporter to test the last ocata point release | |
| 14:11:42 | bauzas | as it includes artom's patch | |
| 14:11:53 | mdbooth | +1 | |
| 14:14:38 | bauzas | dansmith: mmm, we have lots of docs mentioning it's worth doing SIGHUPs for upgrades or mutable config but I don't see how nova-compute service is hooking up this signal :) | |
| 14:14:56 | bauzas | dansmith: since it's not inheriting from oslo.service AFAICT | |
| 14:15:09 | bauzas | and we don't have any signal handling in that code | |
| 14:15:50 | dansmith | bauzas: oh, this reminds me, someone recently asked me about a doc they found that says they could change the log level at runtime | |
| 14:15:54 | dansmith | by SIGHUP | |
| 14:16:11 | dansmith | it clearly was not working and I told them I expected that was oslo documentation, but never circled back | |
| 14:17:03 | bauzas | dansmith: https://bugs.launchpad.net/nova/+bug/1705680 led me investigating and honesly I don't see how the magic can happen | |
| 14:17:04 | openstack | Launchpad bug 1705680 in OpenStack Compute (nova) "nova compute does nothing on receiving sighup signal" [Undecided,New] | |
| 14:17:15 | bauzas | if we were inheriting from oslo.service manager, then OK | |
| 14:17:19 | bauzas | but we're not | |
| 14:17:44 | dansmith | we do process sighup for rpc version pins | |
| 14:18:12 | dansmith | bauzas: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L539 | |
| 14:18:35 | bauzas | dansmith: I saw the hook | |
| 14:18:50 | bauzas | dansmith: but I don't see how we define that hook to be called on a sighup signal | |
| 14:19:03 | dansmith | I think we do get it from oslo.service | |
| 14:19:07 | dansmith | but it's buried pretty deep | |
| 14:19:23 | bauzas | dansmith: that was my assumption | |
| 14:19:24 | bauzas | but | |
| 14:19:29 | bauzas | we don't inherit from it | |
| 14:19:34 | dansmith | we do | |
| 14:19:46 | dansmith | bauzas: https://github.com/openstack/nova/blob/master/nova/service.py#L98-L98 | |
| 14:19:52 | dansmith | bauzas: service is oslo.service there | |
| 14:20:08 | dansmith | L27 | |
| 14:20:11 | bauzas | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L482 | |
| 14:20:22 | bauzas | manager is nova.manager, right? | |
| 14:20:37 | dansmith | yes, but the service hooks Service.reset() to Manager.reset() | |
| 14:20:47 | dansmith | https://github.com/openstack/nova/blob/master/nova/service.py#L289-L289 | |
| 14:21:00 | bauzas | but https://github.com/openstack/nova/blob/master/nova/manager.py#L91 | |
| 14:21:13 | mriedem | dansmith: the mutable config stuff is in oslo.config docs | |
| 14:21:14 | bauzas | dansmith: ooooh, it's fucking cryptic | |
| 14:21:25 | mriedem | https://docs.openstack.org/oslo.config/latest/reference/mutable.html | |
| 14:21:34 | dansmith | bauzas: not really, managers and services have always had this relationship | |
| 14:21:55 | bauzas | I see | |
| 14:21:59 | dansmith | mriedem: oh right and I think we're writing into our default nova config that log level is mutable or something | |
| 14:22:04 | dansmith | because of that | |
| 14:22:56 | mriedem | i thought it was only 'debug' | |
| 14:23:19 | mriedem | https://review.openstack.org/#/c/280851 | |
| 14:23:39 | mriedem | https://review.openstack.org/#/c/254821/ | |
| 14:23:52 | bauzas | mriedem: I was just looking at the mutable-config series | |
| 14:23:53 | dansmith | mriedem: okay I never saw this, but it completely wasn't working | |
| 14:23:55 | mriedem | so default_log_levels config wouldn't use that | |
| 14:24:17 | dansmith | I think it was debug= they were toggling, but maybe not | |
| 14:24:51 | mriedem | https://docs.openstack.org/nova/latest/sample_config.html | |
| 14:24:51 | mriedem | we only have 4 | |
| 14:24:58 | mriedem | if you search for "Note: This option can be changed without restarting." | |
| 14:25:28 | dansmith | wonder if we're supposed to be hooking our sighup handler to oslo.log somehow? | |
| 14:25:37 | bauzas | mriedem: correct, the mutable-config series was mostly still in progress when lxsli left | |
| 14:25:38 | dansmith | unless it registers its own signal handler quietly | |