| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-08 | |||
| 16:41:20 | mriedem | do you have a stacktrace you can put in a paste? | |
| 16:41:26 | ktibi | yep wait plz | |
| 16:41:31 | mriedem | NoMoreFixedIps ? | |
| 16:41:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fix typo in release note https://review.openstack.org/531854 | |
| 16:41:38 | ktibi | yes | |
| 16:41:42 | bauzas | dansmith: I'll also add a follow-up that will try to find the existing mdev 'allocations' (ie. which mdev is used by which instance) at init_host() | |
| 16:41:54 | stephenfin | jaypipes, mriedem: Could one of you push that reno typo fix through? https://review.openstack.org/531854 | |
| 16:42:13 | mriedem | ktibi: can you report a bug? maybe we can build a whitelist of exception types that shouldn't be used to auto-disable a compute | |
| 16:42:23 | dansmith | bauzas: not sure what kernel race you mean, but cool | |
| 16:42:26 | bauzas | fun fact, I also discovered that the nvidia driver can't support the same physical GPU having more than 1 allocated mdev per guest | |
| 16:42:29 | jaypipes | stephenfin: done | |
| 16:42:30 | lyarwood | mdbooth: back sorry | |
| 16:42:37 | stephenfin | jaypipes: Cheers :) | |
| 16:42:40 | jaypipes | np | |
| 16:43:07 | bauzas | dansmith: the fact that when calling sysfs, libvirtd doesn't get the update so it won't change the values of availableInstances and won't show up the newly created mdecs | |
| 16:43:37 | lyarwood | mdbooth: that's going to break iSCSI volumes | |
| 16:43:42 | ktibi | mriedem https://thepasteb.in/p/zmh8qDG2ZYJIZ | |
| 16:43:48 | mdbooth | lyarwood: How? | |
| 16:43:56 | dansmith | bauzas: oh I understand that bug.. is that why you're saying we can't/shouldn't do it in init_host() | |
| 16:43:57 | dansmith | ? | |
| 16:44:08 | dansmith | bauzas: and what's the status of a fix for that issue? | |
| 16:44:15 | lyarwood | mdbooth: we need the volumes connected before we fetch the storage config | |
| 16:44:31 | bauzas | dansmith: the reboot issue or the kernel issue ? | |
| 16:44:38 | lyarwood | mdbooth: because of this wonderful connection_info stashing bs https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/iscsi.py#L67 | |
| 16:44:39 | bauzas | dansmith: for the kernel, I have a race | |
| 16:44:41 | bauzas | oops | |
| 16:44:43 | bauzas | I have a change | |
| 16:44:44 | mdbooth | lyarwood: That would bizarre | |
| 16:44:47 | dansmith | bauzas: the libvirt-does-not-notice-sysfs-changes issue | |
| 16:45:03 | bauzas | dansmith: last patch I just uploaded couple of mins ago | |
| 16:45:07 | ktibi | dansmith and I don't understand your comment, a cloud with "reasonable activity will not suffer from that" ! you think all openstack plateform have 50 instance creation/ min ? | |
| 16:45:15 | mdbooth | lyarwood: Ah, we do that everywhere | |
| 16:45:25 | bauzas | dansmith: for the init_host thingy, I'll try to use statically generated UUIDs | |
| 16:45:30 | dansmith | bauzas: why does this only effect bootstrapping and not our actual allocation for guests? | |
| 16:45:33 | mriedem | ktibi: hmm, that's less specific than i'd like | |
| 16:45:42 | mriedem | ktibi: i thought it was failing when nova created the port | |
| 16:45:46 | mriedem | but it's failing when nova plugs the vif | |
| 16:45:48 | dansmith | bauzas: aren't the uuids stored in the guest xml already? | |
| 16:46:01 | bauzas | dansmith: if the guest XML is not recreated, yes | |
| 16:46:05 | mdbooth | lyarwood: So we need to ensure that in all cases we connect volumes before getting volume info, which makes sense | |
| 16:46:17 | bauzas | but I think I discovered a couple of places where it's dangerous to count on that | |
| 16:46:26 | bauzas | like hard reboot AFAIR | |
| 16:46:32 | dansmith | ktibi: no, I understand a smaller cloud could hit that, but presumably those clouds are a little less concerned about real users launching a DoS, but I'm not arguing that it should be fixed | |
| 16:46:57 | efried | jaypipes I added a summary on https://review.openstack.org/#/c/526539/ | |
| 16:46:58 | dansmith | bauzas: right but on init_host(), we could find all the previously-used uuids | |
| 16:47:08 | jaypipes | efried: danke | |
| 16:47:14 | bauzas | dansmith: yeah, we can try | |
| 16:47:18 | efried | jaypipes Is there anything else needed to get that patch going? | |
| 16:47:26 | dansmith | bauzas: seems like the obvious thing to do, IMHO | |
| 16:47:33 | bauzas | dansmith: my point is, that run can last a while | |
| 16:47:33 | jaypipes | efried: nope. | |
| 16:47:35 | efried | jaypipes (All action items related to that discussion will happen in later/other patches) | |
| 16:47:41 | bauzas | dansmith: anyway, let's try | |
| 16:47:41 | efried | jaypipes Cool. | |
| 16:47:46 | jaypipes | efried: I just finishing up working on n-r-p in allocation candidates) | |
| 16:47:50 | efried | ++ | |
| 16:47:52 | dansmith | bauzas: if they aren't in the guest xml, then they weren't setup properly when we shut down the last time | |
| 16:48:11 | dansmith | bauzas: so if hard reboot blows the xml away, it was broken before reboot, thus init_host isn't making things worse | |
| 16:48:21 | bauzas | dansmith: that's also interesting to see that ovirt folks also had the approach of a deterministic UUID for the mdev | |
| 16:48:31 | bauzas | dansmith: FYI https://mpolednik.github.io/2017/05/21/vfio-mdev/ | |
| 16:48:52 | dansmith | bauzas: I think a deterministic uuid is a bad idea, personally | |
| 16:48:57 | ktibi | dansmith I have a private cloud plateform with ~1500 Vms, it's not very small for me :/ just a user create 200 instances at 4AM ^^ | |
| 16:49:17 | dansmith | ktibi: again, I'm not arguing | |
| 16:49:23 | bauzas | dansmith: I need to go to the doctor for 1 hour, but I'll work tonight | |
| 16:49:26 | mriedem | ktibi: can you find the related failure from the neutron logs? | |
| 16:49:32 | bauzas | let's wrap up that convo later | |
| 16:49:37 | mriedem | ktibi: all nova gets back from neutron is an event saying vif plugging failed | |
| 16:49:40 | dansmith | bauzas: okay | |
| 16:49:45 | mriedem | which doesn't tell us why it failed at all | |
| 16:49:55 | ktibi | dansmith yes no pb ;) just the issue can touch large cloud I think :/ | |
| 16:50:11 | bauzas | dansmith: anyway, the deterministic UUID thingy is separated from the other changes in my series, so if we see it's a non-sense, I'm fine dropping it | |
| 16:50:18 | dansmith | bauzas: ack | |
| 16:50:21 | mdbooth | lyarwood: Ok, that's slightly more complex, but still not a showstopper. We need to pull host volume operations out of _create_domain_and_network, and explicitly call _connect_volume before getting guest xml if required. | |
| 16:50:37 | mdbooth | Or just move it all into _get_guest_storage_config after all | |
| 16:50:38 | ktibi | mriedem I'll try to get logs from neutron | |
| 16:50:49 | mdbooth | Not what I had in mind, but would still be a consolidation | |
| 16:51:51 | mriedem | mdbooth: is this a WIP? https://review.openstack.org/#/c/531852/ | |
| 16:51:57 | mriedem | if so, can you WIPify it? | |
| 16:52:08 | mdbooth | mriedem: Yep, will do | |
| 16:52:12 | mriedem | thanks | |
| 16:52:20 | lyarwood | mdbooth: yeah context down to _get_guest_storage_config and then merge the encryptor stuff into _connect/disconnect_volume | |
| 16:52:42 | mdbooth | lyarwood: And *remove* it from _create_domain_and_network | |
| 16:53:02 | mdbooth | We can already be confident that nothing is relying on that to do volume connection, because it never has | |
| 16:53:11 | lyarwood | mdbooth: in one change? | |
| 16:53:34 | mdbooth | Was just thinking about that. I don't see why not. | |
| 16:53:50 | mdbooth | You might want the swap_volume patch first, though | |
| 16:53:50 | lyarwood | mdbooth: k, I'll reuse the refactor change | |
| 16:53:54 | mdbooth | And lxc | |
| 16:54:02 | mdbooth | My WIP above? | |
| 16:54:04 | lyarwood | ah true | |
| 16:54:11 | lyarwood | mdbooth: no my original | |
| 16:54:14 | mdbooth | If you're going to take it I'll just abandon that | |
| 16:54:17 | ktibi | mdbooth nothing in neutron-server.log :/ 0warning 0 error | |
| 16:54:21 | mdbooth | kk, cool | |
| 16:55:07 | ktibi | oups sorrty for mriedem | |
| 16:58:43 | mriedem | jaypipes: interested if adding an index is worth it for this https://review.openstack.org/#/c/531132/3/nova/db/sqlalchemy/migrate_repo/versions/376_add_migrations_updated_at_index.py - it's adding an index for listing migrations and filtering on updated_at (--changes-since); the query could have other columns/filters in it, so not sure if the index really helps or not | |
| 16:59:16 | jaypipes | mriedem: soon as I'm done with two other series reviews, will review. | |
| 16:59:20 | mriedem | although my guess is if you just did "nova migration-list --changes-since x" it would help | |
| 17:06:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross cell sort support for get_migrations https://review.openstack.org/517273 | |
| 17:06:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add migration db and object pagination support. https://review.openstack.org/514904 | |