Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-08
16:39:55 mdbooth lyarwood: The importance of ^^^ is that the above is where we currently do connect_volume/attach_encryptors separately somewhat deliberately. We fix that, then everything remaining is a bug.
16:40:03 efried jaypipes Ick. But okay. Guess that's not the worst thing in the world.
16:40:09 mriedem ktibi: what is the error you get in the compute? PortLimitExceeded?
16:40:38 mriedem the api should be checking the port quota based on the number of instances and selected network before doing scheduling
16:40:41 bauzas dansmith: thanks for the reviews :)
16:40:46 mriedem which is racy, but still it's a check that happens
16:40:57 bauzas dansmith: I just added a new WIP change that will see the direction for the kernel race
16:40:58 ktibi mriedem ailed to allocate the network(s), not rescheduling
16:41:15 mriedem ktibi: yeah, but wondering what triggers that, a specific exception
16:41:18 ktibi not quota check :/ just no more IP free in the pool
16:41:20 mriedem do you have a stacktrace you can put in a paste?
16:41:26 ktibi yep wait plz
16:41:31 mriedem NoMoreFixedIps ?
16:41:31 openstackgerrit Stephen Finucane proposed openstack/nova master: Fix typo in release note https://review.openstack.org/531854
16:41:38 ktibi yes
16:41:42 bauzas dansmith: I'll also add a follow-up that will try to find the existing mdev 'allocations' (ie. which mdev is used by which instance) at init_host()
16:41:54 stephenfin jaypipes, mriedem: Could one of you push that reno typo fix through? https://review.openstack.org/531854
16:42:13 mriedem ktibi: can you report a bug? maybe we can build a whitelist of exception types that shouldn't be used to auto-disable a compute
16:42:23 dansmith bauzas: not sure what kernel race you mean, but cool
16:42:26 bauzas fun fact, I also discovered that the nvidia driver can't support the same physical GPU having more than 1 allocated mdev per guest
16:42:29 jaypipes stephenfin: done
16:42:30 lyarwood mdbooth: back sorry
16:42:37 stephenfin jaypipes: Cheers :)
16:42:40 jaypipes np
16:43:07 bauzas dansmith: the fact that when calling sysfs, libvirtd doesn't get the update so it won't change the values of availableInstances and won't show up the newly created mdecs
16:43:37 lyarwood mdbooth: that's going to break iSCSI volumes
16:43:42 ktibi mriedem https://thepasteb.in/p/zmh8qDG2ZYJIZ
16:43:48 mdbooth lyarwood: How?
16:43:56 dansmith bauzas: oh I understand that bug.. is that why you're saying we can't/shouldn't do it in init_host()
16:43:57 dansmith ?
16:44:08 dansmith bauzas: and what's the status of a fix for that issue?
16:44:15 lyarwood mdbooth: we need the volumes connected before we fetch the storage config
16:44:31 bauzas dansmith: the reboot issue or the kernel issue ?
16:44:38 lyarwood mdbooth: because of this wonderful connection_info stashing bs https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/iscsi.py#L67
16:44:39 bauzas dansmith: for the kernel, I have a race
16:44:41 bauzas oops
16:44:43 bauzas I have a change
16:44:44 mdbooth lyarwood: That would bizarre
16:44:47 dansmith bauzas: the libvirt-does-not-notice-sysfs-changes issue
16:45:03 bauzas dansmith: last patch I just uploaded couple of mins ago
16:45:07 ktibi dansmith and I don't understand your comment, a cloud with "reasonable activity will not suffer from that" ! you think all openstack plateform have 50 instance creation/ min ?
16:45:15 mdbooth lyarwood: Ah, we do that everywhere
16:45:25 bauzas dansmith: for the init_host thingy, I'll try to use statically generated UUIDs
16:45:30 dansmith bauzas: why does this only effect bootstrapping and not our actual allocation for guests?
16:45:33 mriedem ktibi: hmm, that's less specific than i'd like
16:45:42 mriedem ktibi: i thought it was failing when nova created the port
16:45:46 mriedem but it's failing when nova plugs the vif
16:45:48 dansmith bauzas: aren't the uuids stored in the guest xml already?
16:46:01 bauzas dansmith: if the guest XML is not recreated, yes
16:46:05 mdbooth lyarwood: So we need to ensure that in all cases we connect volumes before getting volume info, which makes sense
16:46:17 bauzas but I think I discovered a couple of places where it's dangerous to count on that
16:46:26 bauzas like hard reboot AFAIR
16:46:32 dansmith ktibi: no, I understand a smaller cloud could hit that, but presumably those clouds are a little less concerned about real users launching a DoS, but I'm not arguing that it should be fixed
16:46:57 efried jaypipes I added a summary on https://review.openstack.org/#/c/526539/
16:46:58 dansmith bauzas: right but on init_host(), we could find all the previously-used uuids
16:47:08 jaypipes efried: danke
16:47:14 bauzas dansmith: yeah, we can try
16:47:18 efried jaypipes Is there anything else needed to get that patch going?
16:47:26 dansmith bauzas: seems like the obvious thing to do, IMHO
16:47:33 bauzas dansmith: my point is, that run can last a while
16:47:33 jaypipes efried: nope.
16:47:35 efried jaypipes (All action items related to that discussion will happen in later/other patches)
16:47:41 bauzas dansmith: anyway, let's try
16:47:41 efried jaypipes Cool.
16:47:46 jaypipes efried: I just finishing up working on n-r-p in allocation candidates)
16:47:50 efried ++
16:47:52 dansmith bauzas: if they aren't in the guest xml, then they weren't setup properly when we shut down the last time
16:48:11 dansmith bauzas: so if hard reboot blows the xml away, it was broken before reboot, thus init_host isn't making things worse
16:48:21 bauzas dansmith: that's also interesting to see that ovirt folks also had the approach of a deterministic UUID for the mdev
16:48:31 bauzas dansmith: FYI https://mpolednik.github.io/2017/05/21/vfio-mdev/
16:48:52 dansmith bauzas: I think a deterministic uuid is a bad idea, personally
16:48:57 ktibi dansmith I have a private cloud plateform with ~1500 Vms, it's not very small for me :/ just a user create 200 instances at 4AM ^^
16:49:17 dansmith ktibi: again, I'm not arguing
16:49:23 bauzas dansmith: I need to go to the doctor for 1 hour, but I'll work tonight
16:49:26 mriedem ktibi: can you find the related failure from the neutron logs?
16:49:32 bauzas let's wrap up that convo later
16:49:37 mriedem ktibi: all nova gets back from neutron is an event saying vif plugging failed
16:49:40 dansmith bauzas: okay
16:49:45 mriedem which doesn't tell us why it failed at all
16:49:55 ktibi dansmith yes no pb ;) just the issue can touch large cloud I think :/
16:50:11 bauzas dansmith: anyway, the deterministic UUID thingy is separated from the other changes in my series, so if we see it's a non-sense, I'm fine dropping it
16:50:18 dansmith bauzas: ack
16:50:21 mdbooth lyarwood: Ok, that's slightly more complex, but still not a showstopper. We need to pull host volume operations out of _create_domain_and_network, and explicitly call _connect_volume before getting guest xml if required.
16:50:37 mdbooth Or just move it all into _get_guest_storage_config after all
16:50:38 ktibi mriedem I'll try to get logs from neutron
16:50:49 mdbooth Not what I had in mind, but would still be a consolidation
16:51:51 mriedem mdbooth: is this a WIP? https://review.openstack.org/#/c/531852/
16:51:57 mriedem if so, can you WIPify it?
16:52:08 mdbooth mriedem: Yep, will do
16:52:12 mriedem thanks
16:52:20 lyarwood mdbooth: yeah context down to _get_guest_storage_config and then merge the encryptor stuff into _connect/disconnect_volume
16:52:42 mdbooth lyarwood: And *remove* it from _create_domain_and_network
16:53:02 mdbooth We can already be confident that nothing is relying on that to do volume connection, because it never has
16:53:11 lyarwood mdbooth: in one change?
16:53:34 mdbooth Was just thinking about that. I don't see why not.
16:53:50 mdbooth You might want the swap_volume patch first, though
16:53:50 lyarwood mdbooth: k, I'll reuse the refactor change
16:53:54 mdbooth And lxc
16:54:02 mdbooth My WIP above?
16:54:04 lyarwood ah true

Earlier   Later