| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-08 | |||
| 16:33:31 | mriedem | edleafe: the test is already there since it comes before it in the series | |
| 16:33:51 | edleafe | mriedem: the regression test really doesn | |
| 16:34:06 | edleafe | doesn't check that the alternate hosts stuff is working as expected | |
| 16:34:16 | edleafe | just that it doesn't blow up :) | |
| 16:34:23 | cdent | efried: I think that sounds right, but as your patient, I'm going to the bath, will bbl | |
| 16:34:28 | mriedem | so you're going to modify the regresssion test to add to it in the resize patch | |
| 16:34:42 | efried | cdent Enjoy | |
| 16:35:29 | edleafe | mriedem: no, more or less copy parts to make a new functional test that's alternate host-specific | |
| 16:35:30 | ktibi | dansmith Hi do you know why consecutive_build_service_disable_threshold default value is 10 ? | |
| 16:35:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406 | |
| 16:35:47 | ktibi | because I found a method for disable all compute service with a simple user :/ | |
| 16:35:57 | matrohon | mriedem: I'll keep on digging on the neutron side first, since not having the ip_allocation field set in port's attributes retrieved by a REST request, when it is explicitly set in the database is not really clean | |
| 16:36:03 | dansmith | ktibi: yes, because I set it to 10 after discussion in the room with a bunch of operators | |
| 16:36:16 | matrohon | mriedem: don't you think? | |
| 16:36:45 | mriedem | matrohon: i agree that None != unset and there are differences in client side behavior potentially, yes | |
| 16:36:46 | dansmith | ktibi: what's the method? might be something we need to exclude from the disable logic if it's not really a fault | |
| 16:36:56 | mriedem | matrohon: but i don't really know how that's all used on the neutron side | |
| 16:37:04 | ktibi | ok, 10 is good but for now, you can fool nova for disable all compute | |
| 16:37:29 | openstackgerrit | Matthew Booth proposed openstack/nova master: Don't connect volumes in _get_guest_storage_config https://review.openstack.org/531852 | |
| 16:37:32 | jaypipes | efried: yeah, that sounds about right to me, too. | |
| 16:37:36 | mdbooth | lyarwood: ^^^ | |
| 16:37:50 | efried | jaypipes Dangit, that still leaves us with orphan-able shared providers. | |
| 16:37:54 | mdbooth | lyarwood: Lets see how that does against tempest, but that change would be extremely backportable | |
| 16:38:04 | mriedem | ktibi: perhaps you'd like to share how users can do that | |
| 16:38:05 | efried | (aside from being asymmetrical) | |
| 16:38:09 | ktibi | dansmith if you spawn multi instances in a network with 0 free IP, nova fail to spawn and increase the trigger. if you deploy ~60 instances, you disable 6 computes | |
| 16:38:10 | dansmith | ktibi: what's the method? | |
| 16:38:17 | matrohon | mriedem: I'll have a look, but AFAIK, routed networks is the only feature really using the ip_allocation field for now | |
| 16:38:53 | ktibi | one user disable my 15 computes with spawn 200 instance on a /28 private network | |
| 16:39:08 | jaypipes | efried: don't worry about that too much. | |
| 16:39:09 | dansmith | ktibi: if you only have 6 computes or are packing tight I guess, but remember it has to be consecutive, so a cloud with reasonable activity will not suffer from that | |
| 16:39:20 | jaypipes | efried: let's focus on the nested providers handling. | |
| 16:39:26 | dansmith | ktibi: but if there's something we can catch about that specific case and not count it in the total, that would make sense | |
| 16:39:27 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: libvirt: Use only existing mdevs if kernel race https://review.openstack.org/531853 | |
| 16:39:38 | jaypipes | efried: orphaned stuff can always be cleaned up by an external perioidic. | |
| 16:39:55 | mdbooth | lyarwood: The importance of ^^^ is that the above is where we currently do connect_volume/attach_encryptors separately somewhat deliberately. We fix that, then everything remaining is a bug. | |
| 16:40:03 | efried | jaypipes Ick. But okay. Guess that's not the worst thing in the world. | |
| 16:40:09 | mriedem | ktibi: what is the error you get in the compute? PortLimitExceeded? | |
| 16:40:38 | mriedem | the api should be checking the port quota based on the number of instances and selected network before doing scheduling | |
| 16:40:41 | bauzas | dansmith: thanks for the reviews :) | |
| 16:40:46 | mriedem | which is racy, but still it's a check that happens | |
| 16:40:57 | bauzas | dansmith: I just added a new WIP change that will see the direction for the kernel race | |
| 16:40:58 | ktibi | mriedem ailed to allocate the network(s), not rescheduling | |
| 16:41:15 | mriedem | ktibi: yeah, but wondering what triggers that, a specific exception | |
| 16:41:18 | ktibi | not quota check :/ just no more IP free in the pool | |
| 16:41:20 | mriedem | do you have a stacktrace you can put in a paste? | |
| 16:41:26 | ktibi | yep wait plz | |
| 16:41:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fix typo in release note https://review.openstack.org/531854 | |
| 16:41:31 | mriedem | NoMoreFixedIps ? | |
| 16:41:38 | ktibi | yes | |
| 16:41:42 | bauzas | dansmith: I'll also add a follow-up that will try to find the existing mdev 'allocations' (ie. which mdev is used by which instance) at init_host() | |
| 16:41:54 | stephenfin | jaypipes, mriedem: Could one of you push that reno typo fix through? https://review.openstack.org/531854 | |
| 16:42:13 | mriedem | ktibi: can you report a bug? maybe we can build a whitelist of exception types that shouldn't be used to auto-disable a compute | |
| 16:42:23 | dansmith | bauzas: not sure what kernel race you mean, but cool | |
| 16:42:26 | bauzas | fun fact, I also discovered that the nvidia driver can't support the same physical GPU having more than 1 allocated mdev per guest | |
| 16:42:29 | jaypipes | stephenfin: done | |
| 16:42:30 | lyarwood | mdbooth: back sorry | |
| 16:42:37 | stephenfin | jaypipes: Cheers :) | |
| 16:42:40 | jaypipes | np | |
| 16:43:07 | bauzas | dansmith: the fact that when calling sysfs, libvirtd doesn't get the update so it won't change the values of availableInstances and won't show up the newly created mdecs | |
| 16:43:37 | lyarwood | mdbooth: that's going to break iSCSI volumes | |
| 16:43:42 | ktibi | mriedem https://thepasteb.in/p/zmh8qDG2ZYJIZ | |
| 16:43:48 | mdbooth | lyarwood: How? | |
| 16:43:56 | dansmith | bauzas: oh I understand that bug.. is that why you're saying we can't/shouldn't do it in init_host() | |
| 16:43:57 | dansmith | ? | |
| 16:44:08 | dansmith | bauzas: and what's the status of a fix for that issue? | |
| 16:44:15 | lyarwood | mdbooth: we need the volumes connected before we fetch the storage config | |
| 16:44:31 | bauzas | dansmith: the reboot issue or the kernel issue ? | |
| 16:44:38 | lyarwood | mdbooth: because of this wonderful connection_info stashing bs https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/iscsi.py#L67 | |
| 16:44:39 | bauzas | dansmith: for the kernel, I have a race | |
| 16:44:41 | bauzas | oops | |
| 16:44:43 | bauzas | I have a change | |
| 16:44:44 | mdbooth | lyarwood: That would bizarre | |
| 16:44:47 | dansmith | bauzas: the libvirt-does-not-notice-sysfs-changes issue | |
| 16:45:03 | bauzas | dansmith: last patch I just uploaded couple of mins ago | |
| 16:45:07 | ktibi | dansmith and I don't understand your comment, a cloud with "reasonable activity will not suffer from that" ! you think all openstack plateform have 50 instance creation/ min ? | |
| 16:45:15 | mdbooth | lyarwood: Ah, we do that everywhere | |
| 16:45:25 | bauzas | dansmith: for the init_host thingy, I'll try to use statically generated UUIDs | |
| 16:45:30 | dansmith | bauzas: why does this only effect bootstrapping and not our actual allocation for guests? | |
| 16:45:33 | mriedem | ktibi: hmm, that's less specific than i'd like | |
| 16:45:42 | mriedem | ktibi: i thought it was failing when nova created the port | |
| 16:45:46 | mriedem | but it's failing when nova plugs the vif | |
| 16:45:48 | dansmith | bauzas: aren't the uuids stored in the guest xml already? | |
| 16:46:01 | bauzas | dansmith: if the guest XML is not recreated, yes | |
| 16:46:05 | mdbooth | lyarwood: So we need to ensure that in all cases we connect volumes before getting volume info, which makes sense | |
| 16:46:17 | bauzas | but I think I discovered a couple of places where it's dangerous to count on that | |
| 16:46:26 | bauzas | like hard reboot AFAIR | |
| 16:46:32 | dansmith | ktibi: no, I understand a smaller cloud could hit that, but presumably those clouds are a little less concerned about real users launching a DoS, but I'm not arguing that it should be fixed | |
| 16:46:57 | efried | jaypipes I added a summary on https://review.openstack.org/#/c/526539/ | |
| 16:46:58 | dansmith | bauzas: right but on init_host(), we could find all the previously-used uuids | |
| 16:47:08 | jaypipes | efried: danke | |
| 16:47:14 | bauzas | dansmith: yeah, we can try | |
| 16:47:18 | efried | jaypipes Is there anything else needed to get that patch going? | |
| 16:47:26 | dansmith | bauzas: seems like the obvious thing to do, IMHO | |
| 16:47:33 | jaypipes | efried: nope. | |
| 16:47:33 | bauzas | dansmith: my point is, that run can last a while | |
| 16:47:35 | efried | jaypipes (All action items related to that discussion will happen in later/other patches) | |
| 16:47:41 | efried | jaypipes Cool. | |
| 16:47:41 | bauzas | dansmith: anyway, let's try | |
| 16:47:46 | jaypipes | efried: I just finishing up working on n-r-p in allocation candidates) | |