| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-24 | |||
| 13:25:34 | mriedem | log_format='%(client_ip)s - - [%(date_time)s] "%(request_line)s" %(status_code)s %(body_length)s %(wall_seconds).6f' | |
| 13:25:34 | mriedem | this is the eventlet wsgi default | |
| 13:26:24 | mriedem | seems odd that we define a default for our config option that's different from the eventlet wsgi default | |
| 13:26:32 | sdague | mriedem: https://github.com/openstack/oslo.service/blob/5a8a712c26cb053cbabf49044e6831ed98a2c2d5/oslo_service/wsgi.py#L184 | |
| 13:26:56 | sdague | mriedem: yeh, well, it's all leaky abstraction | |
| 13:27:02 | sdague | those are eventlet native stanzas | |
| 13:27:12 | mriedem | nova uses it here https://github.com/openstack/nova/blob/master/nova/wsgi.py#L184 | |
| 13:30:27 | sdague | mriedem: ah, I guess we're duping more of oslo.service than I realized, I thought that had been fully factored back out | |
| 13:30:41 | sdague | but, regardless, the current keying are eventlet server keywords | |
| 13:31:02 | mriedem | right, what i'm thinking is if we used the config, | |
| 13:31:14 | mriedem | it would be nice to have it default to None so we could tell if the operator actually cares about overriding it, | |
| 13:31:27 | mriedem | if it's None, and you're using eventlet wsgi, then you get eventlet wsgi defaults for the log format | |
| 13:31:30 | bauzas | mdbooth: do you remember some problems with volume migrations for their type ? https://bugs.launchpad.net/nova/+bug/1705700 | |
| 13:31:31 | openstack | Launchpad bug 1705700 in OpenStack Compute (nova) "live migration does not work after volume migration" [Undecided,New] | |
| 13:31:31 | sdague | mriedem: yeh, very much good point | |
| 13:31:40 | mriedem | if it's None and you're using apache, then you get our hard-coded line | |
| 13:32:01 | sdague | mriedem: we'd also have to normalize the keywords, because things like microversion aren't going to exist where eventlet does this | |
| 13:32:10 | mriedem | right, i'd munge that in with request_line | |
| 13:32:23 | bauzas | mdbooth: tl;dr cinder block volumes seem to be transformed as files | |
| 13:32:38 | mdbooth | bauzas: Looking | |
| 13:33:07 | sdague | https://github.com/eventlet/eventlet/blob/ce72c4378b176cdb0d8b6039023e3ad0026becde/eventlet/wsgi.py#L603-L611 - is where eventlet does it's thing | |
| 13:33:08 | mdbooth | bauzas: That does ring a bell, actually | |
| 13:33:14 | mriedem | sdague: fyi it's also a default value in oslo.service https://github.com/openstack/oslo.service/blob/master/oslo_service/_options.py#L59 | |
| 13:33:19 | sdague | mriedem: yep | |
| 13:33:26 | mdbooth | kashyap: You've fixed ^^^ recently, haven't you? | |
| 13:33:28 | bauzas | mdbooth: yup, I remember some discussion we had for that | |
| 13:33:53 | bauzas | kashyap: https://bugs.launchpad.net/nova/+bug/1705700 | |
| 13:33:54 | openstack | Launchpad bug 1705700 in OpenStack Compute (nova) "live migration does not work after volume migration" [Undecided,New] | |
| 13:34:08 | sdague | mriedem: so, honestly, my inclination is to just say this isn't customizable any more, and only bring that back if people really complain about it | |
| 13:34:16 | mriedem | seems we could deprecate the nova specific wsgi_log_format, although if we want to default it to None we'd then end up relying on the config from oslo.service, which gets messy | |
| 13:34:16 | kashyap | mdbooth: It's the cache value being not set on the Cinder volume post-migration? | |
| 13:34:21 | sdague | because the value in making that site specific seems really weird | |
| 13:34:26 | mdbooth | kashyap: No, different bug | |
| 13:34:33 | kashyap | If so -- https://review.openstack.org/#/c/485752/ -- libvirt: Post-migration, set cache value for Cinder volume(s) | |
| 13:34:38 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform instance.resize_revert notification https://review.openstack.org/396811 | |
| 13:34:46 | kashyap | mdbooth: Oh, is it the one bauzas pointed out just now? | |
| 13:34:53 | mdbooth | kashyap: Yes | |
| 13:35:12 | sdague | mriedem: because it feels like you could do a lot of effort here for 0 people ever doing this thing | |
| 13:35:14 | kashyap | mdbooth: Oh, this one -- I recall it | |
| 13:35:16 | mdbooth | https://bugs.launchpad.net/nova/+bug/1705700 | |
| 13:35:17 | openstack | Launchpad bug 1705700 in OpenStack Compute (nova) "live migration does not work after volume migration" [Undecided,New] | |
| 13:35:31 | mriedem | sdague: yeah i'm asking in the operators channel, but i don't care to bend over backwards for the config option | |
| 13:35:40 | bauzas | mdbooth: looked related but not the same problem | |
| 13:35:46 | bauzas | oops kashyap rather | |
| 13:36:15 | kashyap | mdbooth: Not me (Artom), but I did the triage of it with Eric Blake. The change was simple | |
| 13:36:19 | mdbooth | bauzas: Yeah, I've definitely seen a patch for the file='block device' thing | |
| 13:36:21 | bauzas | claudiub: I'm unclear whether https://bugs.launchpad.net/nova/+bug/1705683 is only for hyper-v driver | |
| 13:36:22 | openstack | Launchpad bug 1705683 in compute-hyperv "vif ports are not unplugged after cold migration" [Undecided,In progress] - Assigned to Alexandru Muresan (amuresan) | |
| 13:36:29 | mriedem | sdague: https://github.com/openstack/nova/commit/c01c852c808970ff649c26a0e52d34b3c2d80b0b | |
| 13:36:53 | kashyap | We need both the flags: VIR_DOMAIN_BLOCK_REBASE_COPY _and_ VIR_DOMAIN_BLOCK_REBASE_COPY_DEV | |
| 13:37:08 | mriedem | sdague: sounds like that was a bug fix at one point or a workaround for a limitation in eventlet | |
| 13:37:23 | mriedem | sdague: and then oslo.service copied it from nova | |
| 13:37:27 | mriedem | https://github.com/openstack/oslo.service/commit/6dd3ad50b6fee79b4cc08fe0830327803d4b2f5b | |
| 13:37:56 | bauzas | mdbooth: if you can find it, you'll get eternal gratitudes | |
| 13:38:17 | mdbooth | bauzas: I'll have a hunt in a bit. Just doing some fun, fun, fun bugzilla diving. | |
| 13:38:19 | bauzas | mdbooth: but I can lookup gerrit for libvirt/volume.py recent changes | |
| 13:38:55 | bauzas | mdbooth: nah, I can try to find so | |
| 13:38:55 | mriedem | sdague: so i'd say we keep them separate and actually deprecate the nova wsgi_log_format option since it's in oslo.service, and doesn't even really seem useful anymore | |
| 13:39:10 | kashyap | mdbooth: Whoops, I think I see the cause: "The issue is caused by the fact that dist type is changed from 'block' to 'file' after volume migration." | |
| 13:39:20 | bauzas | mdbooth: you'll just miss unlimited beer credits | |
| 13:39:22 | kashyap | It should _retain_ 'block' as the disk type post-migration, too. | |
| 13:39:27 | sdague | mriedem: ah, right, because we're going through the python logger the default logger in eventlet is also putting in timestamping | |
| 13:39:44 | bauzas | kashyap: that is the root cause as per the reporter indeed | |
| 13:40:07 | sdague | mriedem: ok, same patch for deprecation, or follow on? | |
| 13:40:12 | bauzas | kashyap: I lean to mark the bug as confirmed, but finding a duplicate or even a bugfix would be awesomeness | |
| 13:40:13 | mriedem | follow on | |
| 13:40:31 | sdague | mriedem: then you want to hit the +A on the current one? | |
| 13:40:41 | kashyap | bauzas: Yeah, have to look for the dupliate. First, trying to wrap my head around another unit test fix. Want to get this out before I'm out for 2-ish weeks starting 01 Aug | |
| 13:40:43 | sdague | it has to +2s, but was waiting for you to come back around | |
| 13:41:00 | mriedem | sdague: i'm goign to push a devstack change that makes nova-api run under eventlet again and depends-on this change to make sure the eventlet stuff is as expected | |
| 13:41:05 | bauzas | kashyap: np, will try to wake up my gerrit search skills | |
| 13:41:06 | kashyap | bauzas: I don't think there's a duplicate. I'm looking at this change for the first time | |
| 13:41:10 | kashyap | s/change/bug/ | |
| 13:41:22 | sdague | mriedem: sure, though the unit tests mostly cover that as well | |
| 13:41:32 | kashyap | bauzas: And the above bug you point to seems caused by this (merged) change: https://review.openstack.org/#/c/465205/ | |
| 13:41:51 | gibi | mriedem: hi! I made some progress on the missing update on the updated_at field. However fix I'm currently proposing might not what we want at the end https://review.openstack.org/#/c/486561/ | |
| 13:42:06 | bauzas | kashyap: was just digging in https://review.openstack.org/#/q/project:openstack/nova+file:%255Enova/virt/libvirt/volume.py | |
| 13:42:46 | gibi | mriedem: I will try to dig deeper in oslo.db as time allows | |
| 13:43:15 | bauzas | kashyap: ok, I'll set the bug as confirmed | |
| 13:43:36 | mriedem | gibi: ok | |
| 13:44:49 | bauzas | mriedem: would it be reasonable to target bug reports as pike-rc-potential if anyone didn't provided a fix yet (ie. not in Progress) ? | |
| 13:45:50 | bauzas | mriedem: having in-progress bugs in the bucket is cool for reviews, but I'd also like to make sure we're like telling to the world that the release is worth getting those unassigned bugs fixed so someone could step up ? | |
| 13:46:07 | mriedem | bauzas: yes | |
| 13:46:31 | bauzas | mriedem: tbc, I'm worried of any bad press of https://bugs.launchpad.net/nova/+bug/1705700 if not fixed by Ocata timeframe | |
| 13:46:32 | openstack | Launchpad bug 1705700 in OpenStack Compute (nova) "live migration does not work after volume migration" [High,Confirmed] | |
| 13:47:04 | claudiub | bauzas: we've only observed that bug on hyper-v. libvirt seems to unplug the vifs on confirm_migration. | |
| 13:48:14 | mriedem | bauzas: is it latent or a regression in pike? | |
| 13:48:27 | bauzas | mriedem: it's Ocata | |
| 13:48:47 | mriedem | so a regression introduced in ocata | |
| 13:48:50 | bauzas | mriedem: but looks like https://review.openstack.org/#/c/465205/ was backported to Ocata | |
| 13:49:17 | bauzas | mriedem: I can try to investigate further | |
| 13:49:35 | bauzas | at least the bugfix above was backported in a point release | |
| 13:49:47 | bauzas | we can ask to test some compute with some older release | |
| 13:50:05 | openstackgerrit | sahid proposed openstack/nova-specs master: virt: provide a mask to selection pCPUs to run emu. threads https://review.openstack.org/486617 | |
| 13:50:34 | edleafe | Scheduler subteam meeting in 10 minutes in #openstack-meeting-alt | |
| 13:51:51 | openstackgerrit | Jianghua Wang proposed openstack/nova-specs master: Support virtual GPU resources https://review.openstack.org/450122 | |
| 14:01:30 | edleafe | Scheduler subteam meeting running now in #openstack-meeting-alt | |
| 14:01:48 | openstackgerrit | Sean Dague proposed openstack/nova master: deprecate ``wsgi_log_format`` config variable https://review.openstack.org/486623 | |
| 14:08:42 | artom | kashyap, mdbooth, you mean | |
| 14:09:18 | artom | Err | |