| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-20 | |||
| 14:29:17 | dansmith | mdbooth: the other processes don't open the log, they inherit it across the fork | |
| 14:30:00 | mdbooth | dansmith: Oh, interesting. However, the locking would still be python thread locking. | |
| 14:30:25 | dansmith | not if logging is locking the file | |
| 14:30:26 | cdent | cfriesen: I’ve been wondering if part of that was due somehow to the doubling stuff | |
| 14:30:53 | mdbooth | Unless the logger is taking and releasing an os lock for every write? | |
| 14:32:17 | sdague | mdbooth: I really think that once you push sufficiently large writes through the python logging buffer, this is just the python behavior | |
| 14:32:25 | sdague | and the only fix is don't do that | |
| 14:33:56 | openstackgerrit | sahid proposed openstack/os-vif master: ovs-hybrid: should permanently keep MAC entries https://review.openstack.org/501132 | |
| 14:35:03 | mdbooth | https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing | |
| 14:35:34 | mdbooth | According to ^^^ in python 3 at least logging doesn't use external locks | |
| 14:36:11 | mdbooth | I guess that would make it a bug in oslo.log | |
| 14:36:40 | openstackgerrit | Merged openstack/nova master: [placement] Unregister the ResourceClass object https://review.openstack.org/502155 | |
| 14:36:50 | stephenfin | sahid: Lovely. +Wd | |
| 14:36:51 | mdbooth | Same for python 2 | |
| 14:37:01 | mdbooth | And I don't see oslo.log importing multiprocessing | |
| 14:37:11 | mdbooth | Well, it does, but it doesn't seem to use it | |
| 14:37:15 | mdbooth | That's pretty weird | |
| 14:38:10 | sdague | mdbooth: I expect that when you don't overrun the python logging natural buffer it just works | |
| 14:38:18 | sdague | and when you do, you get funkiness | |
| 14:38:28 | sdague | and I agree, if you want to fix it, you have to do it down in oslo.log | |
| 14:38:34 | mdbooth | sdague: We log exceptions, though, which are kinda arbitrarily large | |
| 14:38:42 | mdbooth | I don't think we want to stop doing that | |
| 14:39:02 | sdague | mdbooth: we do, but we've apparently been lucky thus far | |
| 14:39:03 | mdbooth | I'll move the bug to oslo.log and mention the multiprocess thing | |
| 14:39:50 | mdbooth | Assuming, that is, we don't want to open separate log files for conductor workers? | |
| 14:40:05 | mdbooth | Because that would be a nova fix, and possibly useful in its own right | |
| 14:40:42 | sdague | mdbooth: no, because that problem is equally theoretically a problem for all the other services as well | |
| 14:40:47 | dansmith | I definitely don't want separate logs for conductor workers | |
| 14:40:47 | sdague | except nova-compute | |
| 14:40:51 | dansmith | because..holy crap | |
| 14:40:54 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-traits master: Updated from global requirements https://review.openstack.org/503646 | |
| 14:40:57 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708 | |
| 14:41:08 | mdbooth | dansmith: Merged logs, ftw! ;) | |
| 14:41:09 | sdague | and it totally would wreck all the log injest systems people have | |
| 14:41:29 | mdbooth | True. | |
| 14:41:34 | mdbooth | oslo.log it is | |
| 14:43:30 | jamespage | mriedem: working a fix now | |
| 14:43:38 | cfriesen | cdent: that does seem to be the most likely suspect. points to a gap in our testing. | |
| 14:45:25 | openstackgerrit | Merged openstack/nova master: Update docs for _destroy_evacuated_instances https://review.openstack.org/500144 | |
| 14:46:00 | mriedem | jamespage: i'm testing sdague's idea too | |
| 14:46:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: check qemu version when calling qemu-img info https://review.openstack.org/505673 | |
| 14:46:34 | openstackgerrit | Elod Illes proposed openstack/nova master: Add instance.interface_attach notification https://review.openstack.org/503089 | |
| 14:46:58 | cdent | cfriesen: gaps in testing is a bit of a trend, but gibi is fixing it ;) | |
| 14:47:51 | openstackgerrit | Chris Dent proposed openstack/nova master: Add functional test for two-cell scheduler behaviors https://review.openstack.org/452006 | |
| 14:47:58 | mriedem | jamespage: testing here https://review.openstack.org/#/c/505674/ | |
| 14:48:01 | sdague | mriedem: you got a devstack patch depends on that? | |
| 14:48:16 | mriedem | yes ^ | |
| 14:48:47 | sdague | ah cool | |
| 14:49:27 | gibi | cdent: do you mean I should add a test case which boots multiple instances with a single boot command then try to migrate them? ;) | |
| 14:49:56 | cdent | I meant you were fixing the trend more generally but if you’re feeling motivated :) | |
| 14:50:24 | dansmith | mriedem: if you're okay with it I'm just going to fast approve all these unregister patches as they're just mechanical search/replace: https://review.openstack.org/#/c/502157/5 | |
| 14:51:24 | openstackgerrit | Merged openstack/nova master: Add @targets_cell for live_migrate_instance method in conductor https://review.openstack.org/503601 | |
| 14:51:56 | mriedem | dansmith: i can go through them quick | |
| 14:52:14 | mriedem | cfriesen: re that ML thread, he's disabling a host and live migrating off the source host - does he mean evacuating off the source host? | |
| 14:53:14 | gibi | cdent: at least I made TODO on my desk about it but I'm not promising anything :) | |
| 14:54:11 | dansmith | mriedem: okay | |
| 14:54:34 | mriedem | dansmith: cdent: question in https://review.openstack.org/#/c/502157/5//COMMIT_MSG but just to make sure i know what we're doing in the series | |
| 14:55:41 | dansmith | mriedem: I'm not sure I understand what you're asking | |
| 14:56:07 | dansmith | oh I see, | |
| 14:56:16 | dansmith | because there weren't any actual object references anywhere | |
| 14:56:21 | dansmith | in that patch specifically | |
| 14:57:10 | cdent | yeah, that’s a pasto | |
| 14:57:15 | mriedem | yeah, ok | |
| 14:57:17 | openstackgerrit | Eric Berglund proposed openstack/nova master: Add PowerVM hypervisor configuration doc https://review.openstack.org/505665 | |
| 14:57:19 | mriedem | alright moving on | |
| 14:58:05 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-traits master: Updated from global requirements https://review.openstack.org/503646 | |
| 14:58:09 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708 | |
| 14:58:51 | mriedem | cfriesen: on that migrate thread, i think in pike ed added the new stuff to pass the number of instances from the request spec (something like that) to the scheduler, so it could be why it's thinking it wants to migrate all 10 at once, from the persisted request spec | |
| 14:58:56 | mriedem | might be good for edleafe to look at that | |
| 14:59:40 | mriedem | https://review.openstack.org/#/c/465171/ | |
| 15:00:47 | mriedem | but this passes just the single instance being live migrated https://review.openstack.org/#/c/465171/11/nova/conductor/tasks/live_migrate.py | |
| 15:05:47 | sahid | melwitt: hum i do not understand your comment | |
| 15:05:49 | mriedem | cfriesen: this looks like a latent issue https://github.com/openstack/nova/blob/8a386b055c82df67092a1abc683e7225ef80671e/nova/scheduler/filter_scheduler.py#L81 | |
| 15:05:55 | sahid | https://review.openstack.org/#/c/334614/ | |
| 15:05:59 | mriedem | we're checking the number of instances from the request spec | |
| 15:06:05 | mriedem | which is persisted when we created the instance with multi-create | |
| 15:06:11 | mriedem | and using that during the live migration of the single instance | |
| 15:06:17 | cdent | bauzas: are you still invested in keeping your -1 on https://review.openstack.org/#/c/419502/ | |
| 15:06:20 | sahid | not sure if you want me to comment an other time or if we can discuss about it | |
| 15:06:36 | mriedem | cfriesen: "num_instances = spec_obj.num_instances" should probably be changed to "num_instances = len(instance_uuids)" now | |
| 15:07:06 | mriedem | cfriesen: i'm not subscribed to the general list but you might ask them to change that line and see if it works | |
| 15:07:32 | bauzas | cdent: I think my concern is still valid | |
| 15:07:42 | bauzas | cdent: renaming an AZ is having an user impact | |
| 15:08:18 | dansmith | bauzas: cdent dear god | |
| 15:08:46 | mriedem | you should just not be able to rename an az while there are instances in that az | |
| 15:08:47 | cdent | bauzas: I was checking because as far as I could read the bug report, the “bad” solution was considered good enough | |
| 15:08:47 | dansmith | that operation could be across cells, and across a loooot of instances. It could fail in the middle leaving things highly confusing | |
| 15:08:52 | dansmith | mriedem: ++ | |
| 15:09:08 | bauzas | mriedem: that was my point | |
| 15:09:10 | mriedem | if you want to rename the az, migrate the instances out of it | |
| 15:09:23 | mriedem | but i'm not sure if that's possible? | |
| 15:09:31 | mriedem | unless you use the force flag... | |
| 15:09:54 | mriedem | but if the operator forces a live migration of an instance from az1 to az2, do we even update the instance.availability_zone field? | |
| 15:10:03 | cdent | I think the implementation was following sean’s advice: https://bugs.launchpad.net/nova/+bug/1378904/comments/5 | |
| 15:10:04 | openstack | Launchpad bug 1378904 in OpenStack Compute (nova) "renaming availability zone doesn't modify host's availability zone" [Low,In progress] - Assigned to Radoslav Gerganov (rgerganov) | |
| 15:10:21 | cdent | but if there’s a bigger problem than that, would be great to see those problems on the review | |
| 15:10:43 | bauzas | mriedem: I think migrating a VM requires some kind of discussion between the operator and the end user | |
| 15:10:52 | bauzas | not something magical per so | |
| 15:10:53 | bauzas | se | |
| 15:11:03 | dansmith | migrating across AZs without the user's input is dangerous too | |