| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-20 | |||
| 14:26:53 | sdague | because I thought we wanted image to be distinct | |
| 14:27:10 | sdague | mdbooth: I'm not sure what help a nova bug does here, it's really not fixable in nova | |
| 14:27:11 | cfriesen | is anyone aware of a scheduling issue in Pike when rescheduling instances that were originally booted as part of a multi-instance boot? There's a thread "Pike NOVA Disable and Live Migrate all instances" on the openstack list that seems to indicate a bug. | |
| 14:27:22 | openstackgerrit | Merged openstack/nova master: conf: Rename two VNC options https://review.openstack.org/498387 | |
| 14:27:36 | dansmith | mdbooth: python logging should be locking the fd, which is opened before the fork | |
| 14:27:49 | mdbooth | sdague: Well I found it in Nova, and it's definitely a thing. There's likely to be a better place to move it. | |
| 14:28:12 | mriedem | mdbooth: if you found it in nova, it's also then probably an issue in all other services | |
| 14:28:20 | mriedem | if it's logging related | |
| 14:28:42 | mdbooth | dansmith: That kind of locking could only work by preventing other conductors from opening the log file at all. | |
| 14:28:48 | openstackgerrit | Eric Berglund proposed openstack/nova master: Add PowerVM hypervisor configuration doc https://review.openstack.org/505665 | |
| 14:28:59 | mdbooth | Which would mean that it would block all conductors beyond the first. | |
| 14:29:17 | dansmith | mdbooth: the other processes don't open the log, they inherit it across the fork | |
| 14:30:00 | mdbooth | dansmith: Oh, interesting. However, the locking would still be python thread locking. | |
| 14:30:25 | dansmith | not if logging is locking the file | |
| 14:30:26 | cdent | cfriesen: I’ve been wondering if part of that was due somehow to the doubling stuff | |
| 14:30:53 | mdbooth | Unless the logger is taking and releasing an os lock for every write? | |
| 14:32:17 | sdague | mdbooth: I really think that once you push sufficiently large writes through the python logging buffer, this is just the python behavior | |
| 14:32:25 | sdague | and the only fix is don't do that | |
| 14:33:56 | openstackgerrit | sahid proposed openstack/os-vif master: ovs-hybrid: should permanently keep MAC entries https://review.openstack.org/501132 | |
| 14:35:03 | mdbooth | https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing | |
| 14:35:34 | mdbooth | According to ^^^ in python 3 at least logging doesn't use external locks | |
| 14:36:11 | mdbooth | I guess that would make it a bug in oslo.log | |
| 14:36:40 | openstackgerrit | Merged openstack/nova master: [placement] Unregister the ResourceClass object https://review.openstack.org/502155 | |
| 14:36:50 | stephenfin | sahid: Lovely. +Wd | |
| 14:36:51 | mdbooth | Same for python 2 | |
| 14:37:01 | mdbooth | And I don't see oslo.log importing multiprocessing | |
| 14:37:11 | mdbooth | Well, it does, but it doesn't seem to use it | |
| 14:37:15 | mdbooth | That's pretty weird | |
| 14:38:10 | sdague | mdbooth: I expect that when you don't overrun the python logging natural buffer it just works | |
| 14:38:18 | sdague | and when you do, you get funkiness | |
| 14:38:28 | sdague | and I agree, if you want to fix it, you have to do it down in oslo.log | |
| 14:38:34 | mdbooth | sdague: We log exceptions, though, which are kinda arbitrarily large | |
| 14:38:42 | mdbooth | I don't think we want to stop doing that | |
| 14:39:02 | sdague | mdbooth: we do, but we've apparently been lucky thus far | |
| 14:39:03 | mdbooth | I'll move the bug to oslo.log and mention the multiprocess thing | |
| 14:39:50 | mdbooth | Assuming, that is, we don't want to open separate log files for conductor workers? | |
| 14:40:05 | mdbooth | Because that would be a nova fix, and possibly useful in its own right | |
| 14:40:42 | sdague | mdbooth: no, because that problem is equally theoretically a problem for all the other services as well | |
| 14:40:47 | sdague | except nova-compute | |
| 14:40:47 | dansmith | I definitely don't want separate logs for conductor workers | |
| 14:40:51 | dansmith | because..holy crap | |
| 14:40:54 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-traits master: Updated from global requirements https://review.openstack.org/503646 | |
| 14:40:57 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708 | |
| 14:41:08 | mdbooth | dansmith: Merged logs, ftw! ;) | |
| 14:41:09 | sdague | and it totally would wreck all the log injest systems people have | |
| 14:41:29 | mdbooth | True. | |
| 14:41:34 | mdbooth | oslo.log it is | |
| 14:43:30 | jamespage | mriedem: working a fix now | |
| 14:43:38 | cfriesen | cdent: that does seem to be the most likely suspect. points to a gap in our testing. | |
| 14:45:25 | openstackgerrit | Merged openstack/nova master: Update docs for _destroy_evacuated_instances https://review.openstack.org/500144 | |
| 14:46:00 | mriedem | jamespage: i'm testing sdague's idea too | |
| 14:46:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: check qemu version when calling qemu-img info https://review.openstack.org/505673 | |
| 14:46:34 | openstackgerrit | Elod Illes proposed openstack/nova master: Add instance.interface_attach notification https://review.openstack.org/503089 | |
| 14:46:58 | cdent | cfriesen: gaps in testing is a bit of a trend, but gibi is fixing it ;) | |
| 14:47:51 | openstackgerrit | Chris Dent proposed openstack/nova master: Add functional test for two-cell scheduler behaviors https://review.openstack.org/452006 | |
| 14:47:58 | mriedem | jamespage: testing here https://review.openstack.org/#/c/505674/ | |
| 14:48:01 | sdague | mriedem: you got a devstack patch depends on that? | |
| 14:48:16 | mriedem | yes ^ | |
| 14:48:47 | sdague | ah cool | |
| 14:49:27 | gibi | cdent: do you mean I should add a test case which boots multiple instances with a single boot command then try to migrate them? ;) | |
| 14:49:56 | cdent | I meant you were fixing the trend more generally but if you’re feeling motivated :) | |
| 14:50:24 | dansmith | mriedem: if you're okay with it I'm just going to fast approve all these unregister patches as they're just mechanical search/replace: https://review.openstack.org/#/c/502157/5 | |
| 14:51:24 | openstackgerrit | Merged openstack/nova master: Add @targets_cell for live_migrate_instance method in conductor https://review.openstack.org/503601 | |
| 14:51:56 | mriedem | dansmith: i can go through them quick | |
| 14:52:14 | mriedem | cfriesen: re that ML thread, he's disabling a host and live migrating off the source host - does he mean evacuating off the source host? | |
| 14:53:14 | gibi | cdent: at least I made TODO on my desk about it but I'm not promising anything :) | |
| 14:54:11 | dansmith | mriedem: okay | |
| 14:54:34 | mriedem | dansmith: cdent: question in https://review.openstack.org/#/c/502157/5//COMMIT_MSG but just to make sure i know what we're doing in the series | |
| 14:55:41 | dansmith | mriedem: I'm not sure I understand what you're asking | |
| 14:56:07 | dansmith | oh I see, | |
| 14:56:16 | dansmith | because there weren't any actual object references anywhere | |
| 14:56:21 | dansmith | in that patch specifically | |
| 14:57:10 | cdent | yeah, that’s a pasto | |
| 14:57:15 | mriedem | yeah, ok | |
| 14:57:17 | openstackgerrit | Eric Berglund proposed openstack/nova master: Add PowerVM hypervisor configuration doc https://review.openstack.org/505665 | |
| 14:57:19 | mriedem | alright moving on | |
| 14:58:05 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-traits master: Updated from global requirements https://review.openstack.org/503646 | |
| 14:58:09 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708 | |
| 14:58:51 | mriedem | cfriesen: on that migrate thread, i think in pike ed added the new stuff to pass the number of instances from the request spec (something like that) to the scheduler, so it could be why it's thinking it wants to migrate all 10 at once, from the persisted request spec | |
| 14:58:56 | mriedem | might be good for edleafe to look at that | |
| 14:59:40 | mriedem | https://review.openstack.org/#/c/465171/ | |
| 15:00:47 | mriedem | but this passes just the single instance being live migrated https://review.openstack.org/#/c/465171/11/nova/conductor/tasks/live_migrate.py | |
| 15:05:47 | sahid | melwitt: hum i do not understand your comment | |
| 15:05:49 | mriedem | cfriesen: this looks like a latent issue https://github.com/openstack/nova/blob/8a386b055c82df67092a1abc683e7225ef80671e/nova/scheduler/filter_scheduler.py#L81 | |
| 15:05:55 | sahid | https://review.openstack.org/#/c/334614/ | |
| 15:05:59 | mriedem | we're checking the number of instances from the request spec | |
| 15:06:05 | mriedem | which is persisted when we created the instance with multi-create | |
| 15:06:11 | mriedem | and using that during the live migration of the single instance | |
| 15:06:17 | cdent | bauzas: are you still invested in keeping your -1 on https://review.openstack.org/#/c/419502/ | |
| 15:06:20 | sahid | not sure if you want me to comment an other time or if we can discuss about it | |
| 15:06:36 | mriedem | cfriesen: "num_instances = spec_obj.num_instances" should probably be changed to "num_instances = len(instance_uuids)" now | |
| 15:07:06 | mriedem | cfriesen: i'm not subscribed to the general list but you might ask them to change that line and see if it works | |
| 15:07:32 | bauzas | cdent: I think my concern is still valid | |
| 15:07:42 | bauzas | cdent: renaming an AZ is having an user impact | |
| 15:08:18 | dansmith | bauzas: cdent dear god | |
| 15:08:46 | mriedem | you should just not be able to rename an az while there are instances in that az | |
| 15:08:47 | dansmith | that operation could be across cells, and across a loooot of instances. It could fail in the middle leaving things highly confusing | |
| 15:08:47 | cdent | bauzas: I was checking because as far as I could read the bug report, the “bad” solution was considered good enough | |
| 15:08:52 | dansmith | mriedem: ++ | |
| 15:09:08 | bauzas | mriedem: that was my point | |