Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-08
16:42:13 mriedem ktibi: can you report a bug? maybe we can build a whitelist of exception types that shouldn't be used to auto-disable a compute
16:42:23 dansmith bauzas: not sure what kernel race you mean, but cool
16:42:26 bauzas fun fact, I also discovered that the nvidia driver can't support the same physical GPU having more than 1 allocated mdev per guest
16:42:29 jaypipes stephenfin: done
16:42:30 lyarwood mdbooth: back sorry
16:42:37 stephenfin jaypipes: Cheers :)
16:42:40 jaypipes np
16:43:07 bauzas dansmith: the fact that when calling sysfs, libvirtd doesn't get the update so it won't change the values of availableInstances and won't show up the newly created mdecs
16:43:37 lyarwood mdbooth: that's going to break iSCSI volumes
16:43:42 ktibi mriedem https://thepasteb.in/p/zmh8qDG2ZYJIZ
16:43:48 mdbooth lyarwood: How?
16:43:56 dansmith bauzas: oh I understand that bug.. is that why you're saying we can't/shouldn't do it in init_host()
16:43:57 dansmith ?
16:44:08 dansmith bauzas: and what's the status of a fix for that issue?
16:44:15 lyarwood mdbooth: we need the volumes connected before we fetch the storage config
16:44:31 bauzas dansmith: the reboot issue or the kernel issue ?
16:44:38 lyarwood mdbooth: because of this wonderful connection_info stashing bs https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/iscsi.py#L67
16:44:39 bauzas dansmith: for the kernel, I have a race
16:44:41 bauzas oops
16:44:43 bauzas I have a change
16:44:44 mdbooth lyarwood: That would bizarre
16:44:47 dansmith bauzas: the libvirt-does-not-notice-sysfs-changes issue
16:45:03 bauzas dansmith: last patch I just uploaded couple of mins ago
16:45:07 ktibi dansmith and I don't understand your comment, a cloud with "reasonable activity will not suffer from that" ! you think all openstack plateform have 50 instance creation/ min ?
16:45:15 mdbooth lyarwood: Ah, we do that everywhere
16:45:25 bauzas dansmith: for the init_host thingy, I'll try to use statically generated UUIDs
16:45:30 dansmith bauzas: why does this only effect bootstrapping and not our actual allocation for guests?
16:45:33 mriedem ktibi: hmm, that's less specific than i'd like
16:45:42 mriedem ktibi: i thought it was failing when nova created the port
16:45:46 mriedem but it's failing when nova plugs the vif
16:45:48 dansmith bauzas: aren't the uuids stored in the guest xml already?
16:46:01 bauzas dansmith: if the guest XML is not recreated, yes
16:46:05 mdbooth lyarwood: So we need to ensure that in all cases we connect volumes before getting volume info, which makes sense
16:46:17 bauzas but I think I discovered a couple of places where it's dangerous to count on that
16:46:26 bauzas like hard reboot AFAIR
16:46:32 dansmith ktibi: no, I understand a smaller cloud could hit that, but presumably those clouds are a little less concerned about real users launching a DoS, but I'm not arguing that it should be fixed
16:46:57 efried jaypipes I added a summary on https://review.openstack.org/#/c/526539/
16:46:58 dansmith bauzas: right but on init_host(), we could find all the previously-used uuids
16:47:08 jaypipes efried: danke
16:47:14 bauzas dansmith: yeah, we can try
16:47:18 efried jaypipes Is there anything else needed to get that patch going?
16:47:26 dansmith bauzas: seems like the obvious thing to do, IMHO
16:47:33 jaypipes efried: nope.
16:47:33 bauzas dansmith: my point is, that run can last a while
16:47:35 efried jaypipes (All action items related to that discussion will happen in later/other patches)
16:47:41 efried jaypipes Cool.
16:47:41 bauzas dansmith: anyway, let's try
16:47:46 jaypipes efried: I just finishing up working on n-r-p in allocation candidates)
16:47:50 efried ++
16:47:52 dansmith bauzas: if they aren't in the guest xml, then they weren't setup properly when we shut down the last time
16:48:11 dansmith bauzas: so if hard reboot blows the xml away, it was broken before reboot, thus init_host isn't making things worse
16:48:21 bauzas dansmith: that's also interesting to see that ovirt folks also had the approach of a deterministic UUID for the mdev
16:48:31 bauzas dansmith: FYI https://mpolednik.github.io/2017/05/21/vfio-mdev/
16:48:52 dansmith bauzas: I think a deterministic uuid is a bad idea, personally
16:48:57 ktibi dansmith I have a private cloud plateform with ~1500 Vms, it's not very small for me :/ just a user create 200 instances at 4AM ^^
16:49:17 dansmith ktibi: again, I'm not arguing
16:49:23 bauzas dansmith: I need to go to the doctor for 1 hour, but I'll work tonight
16:49:26 mriedem ktibi: can you find the related failure from the neutron logs?
16:49:32 bauzas let's wrap up that convo later
16:49:37 mriedem ktibi: all nova gets back from neutron is an event saying vif plugging failed
16:49:40 dansmith bauzas: okay
16:49:45 mriedem which doesn't tell us why it failed at all
16:49:55 ktibi dansmith yes no pb ;) just the issue can touch large cloud I think :/
16:50:11 bauzas dansmith: anyway, the deterministic UUID thingy is separated from the other changes in my series, so if we see it's a non-sense, I'm fine dropping it
16:50:18 dansmith bauzas: ack
16:50:21 mdbooth lyarwood: Ok, that's slightly more complex, but still not a showstopper. We need to pull host volume operations out of _create_domain_and_network, and explicitly call _connect_volume before getting guest xml if required.
16:50:37 mdbooth Or just move it all into _get_guest_storage_config after all
16:50:38 ktibi mriedem I'll try to get logs from neutron
16:50:49 mdbooth Not what I had in mind, but would still be a consolidation
16:51:51 mriedem mdbooth: is this a WIP? https://review.openstack.org/#/c/531852/
16:51:57 mriedem if so, can you WIPify it?
16:52:08 mdbooth mriedem: Yep, will do
16:52:12 mriedem thanks
16:52:20 lyarwood mdbooth: yeah context down to _get_guest_storage_config and then merge the encryptor stuff into _connect/disconnect_volume
16:52:42 mdbooth lyarwood: And *remove* it from _create_domain_and_network
16:53:02 mdbooth We can already be confident that nothing is relying on that to do volume connection, because it never has
16:53:11 lyarwood mdbooth: in one change?
16:53:34 mdbooth Was just thinking about that. I don't see why not.
16:53:50 lyarwood mdbooth: k, I'll reuse the refactor change
16:53:50 mdbooth You might want the swap_volume patch first, though
16:53:54 mdbooth And lxc
16:54:02 mdbooth My WIP above?
16:54:04 lyarwood ah true
16:54:11 lyarwood mdbooth: no my original
16:54:14 mdbooth If you're going to take it I'll just abandon that
16:54:17 ktibi mdbooth nothing in neutron-server.log :/ 0warning 0 error
16:54:21 mdbooth kk, cool
16:55:07 ktibi oups sorrty for mriedem
16:58:43 mriedem jaypipes: interested if adding an index is worth it for this https://review.openstack.org/#/c/531132/3/nova/db/sqlalchemy/migrate_repo/versions/376_add_migrations_updated_at_index.py - it's adding an index for listing migrations and filtering on updated_at (--changes-since); the query could have other columns/filters in it, so not sure if the index really helps or not
16:59:16 jaypipes mriedem: soon as I'm done with two other series reviews, will review.
16:59:20 mriedem although my guess is if you just did "nova migration-list --changes-since x" it would help
17:06:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Add migration db and object pagination support. https://review.openstack.org/514904
17:06:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Add cross cell sort support for get_migrations https://review.openstack.org/517273
17:06:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406
17:06:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Add index(updated_at) on migrations table. https://review.openstack.org/531132
17:06:51 stephenfin ralonsoh: Can we merge [1], given that [2] isn't working yet? [1] https://review.openstack.org/#/c/484386/ [2] https://review.openstack.org/#/c/530961/
17:08:27 mriedem dtantsur: re: CI testing for ironic+bfv https://review.openstack.org/#/c/468353/ - is there at least an existing job that ironic uses for this? if so, we can add that to nova's experimental queue
17:09:46 dtantsur mriedem: yep ironic-tempest-dsvm-bfv
17:14:07 dtantsur wanna me propose a patch?
17:14:14 efried jaypipes (and another core, maybe stephenfin or bauzas?) would you please add https://review.openstack.org/#/c/518633/ to your list? The patch above that is already +W, and it'll be nice to have for your nrp test case (https://review.openstack.org/#/c/531443/)

Earlier   Later