| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-03 | |||
| 15:28:19 | stephenfin | ta | |
| 15:30:55 | dansmith | stephenfin: if it were the db id we wouldn't be able to distinguish one instance over another across cells | |
| 15:31:13 | mriedem | we also don't expose the server primary key id out of the rest api | |
| 15:31:57 | sean-k-mooney | impliying that if it was teh primary key id that neutron would not be able to use it in api queries | |
| 15:32:30 | sean-k-mooney | we did expose the hypervior primary key although i think that was by mistake. | |
| 15:32:43 | mriedem | it wasn't by mistake for hypervisors, | |
| 15:32:51 | mriedem | they originally didn't have uuids | |
| 15:32:57 | sean-k-mooney | ah ok | |
| 15:32:57 | mriedem | same with services and lots of other things | |
| 15:57:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Cache security group driver https://review.opendev.org/697122 | |
| 16:26:31 | dansmith | anybody want to +W this so Sundar can just rebase on master? https://review.opendev.org/#/c/695985/ | |
| 16:26:41 | dansmith | s/rebase/rebase the cyborg stuff/ | |
| 16:27:09 | mriedem | gibi: i replied to your questions in https://review.opendev.org/#/c/637058/, thanks. i'll stack a change on top of the series to see what replacing that setup_networks_on_host hack would look like if we just implement port binding deleting in cleanup_instance_network_on_host | |
| 16:28:50 | mriedem | dansmith: looking | |
| 16:30:06 | gibi | mriedem: ack, thanks | |
| 16:46:24 | openstackgerrit | Thierry Carrez proposed openstack/nova master: Remove unused rootwrap filters https://review.opendev.org/697134 | |
| 17:36:14 | dansmith | eandersson: what perf improvements in nova specifically have shifted your scale focus away from nova? | |
| 17:40:09 | efried | dansmith: Would you please have another look at the vTPM spec. I'd like to get your +2 before you bugger off til 2020. https://review.opendev.org/#/c/686804/ | |
| 17:40:51 | dansmith | efried: I've skimmed it.. tbh, I'm really like -0.9 on it, so if you want me to review it, it's probably going to be not helpful to your effort | |
| 17:41:21 | dansmith | I'm not sure I see the benefit outweighing the nightmare of users trying to predict the behavior | |
| 17:41:46 | dansmith | I was thinking it was required for secure boot, but talking with folks I realize it's not, so ... it's just hard to justify I think | |
| 17:45:09 | sean-k-mooney | for what its worth the out of tree hyperv driver support vtpm for 3-4 years already | |
| 17:45:36 | sean-k-mooney | its part fo there sheilded vms thing but i have no idea how they solved any of the issues with move operations | |
| 17:45:49 | efried | did they? | |
| 17:46:21 | sean-k-mooney | yep ill get the link. did i not send this to you already | |
| 17:46:27 | dansmith | sean-k-mooney: given how hyperv moves things I expect it's tracked by the hypervisor and so it just works, but I could be wrong | |
| 17:46:54 | sean-k-mooney | https://github.com/openstack/compute-hyperv/commit/f37ce8b6bb0eb88a367239698ba7c3df3b64db38 | |
| 17:47:01 | dansmith | the question is more about how things like snapshot works | |
| 17:47:10 | sean-k-mooney | they have an os_vtpm image property | |
| 17:50:00 | sean-k-mooney | they must have patches against nova because the ImageMetaProps object does not accept os_vtpm or os_shielded_vm | |
| 17:53:13 | dansmith | looks like maybe they're doing some mangling of instance.hostname to reference specific keys or something? | |
| 17:53:24 | dansmith | and they're illegally stashing their own stuff in insance.metadata | |
| 17:53:35 | dansmith | so all manner of hacks in that implementation | |
| 17:53:43 | sean-k-mooney | yep | |
| 17:53:55 | sean-k-mooney | and none of this is supported in the in tree one | |
| 17:56:42 | sean-k-mooney | im looking at there snapshot function now https://github.com/openstack/compute-hyperv/blob/cb203978f262f31790592b2a0692fc2acaaef33d/compute_hyperv/nova/snapshotops.py#L66 but i think its just of the image so they dont snapshot the tpm? | |
| 17:57:32 | dansmith | I dunno, but since they've already broken many rules of user interaction, looking further into how they handle it doesn't seem like it would help guide us | |
| 17:57:58 | sean-k-mooney | it proably wont but it looks like they just ignored it | |
| 17:58:27 | sean-k-mooney | so i guess there stance was the tpm state would not be resoreted if you restored form a snapshot | |
| 17:58:39 | sean-k-mooney | the same way non root disks are not restored | |
| 17:59:40 | dansmith | or there is magic in the hostname mangling stuff so that if you recreate an instance with the right magic hostname, it will re-get the tpm stored in the hypervisor? | |
| 18:00:39 | dansmith | given that they basically implement features by bounty in that out of tree driver, I expect they implemented just the part of the solution that would suffice for the one customer asking for it | |
| 18:01:26 | sean-k-mooney | i dont think they are manageling the path in the metatada but yes i would guess the custoemr did not ask for snapshotiing so they did not add it | |
| 18:02:54 | efried | To me, the only severe issue is evacuate. The rest should be easy to understand with proper documentation. | |
| 18:02:59 | sean-k-mooney | i think they were using the vtpm soly for disk encryption | |
| 18:03:19 | efried | "vTPM is tied to your instance. Don't expect to be able to snapshot and clone it." | |
| 18:04:13 | efried | "You can back up and restore as long as you use 'backup'." | |
| 18:04:44 | sean-k-mooney | ? you use backup | |
| 18:05:26 | sean-k-mooney | you mean as long as you rebuild form a snapshot and dont launch an new instance right | |
| 18:05:46 | efried | No. I mean as long as you use rebuild from an image created via the createBackup server action | |
| 18:06:07 | sean-k-mooney | right but does that not just create a shapshot | |
| 18:06:11 | efried | 'snapshot' isn't a server action. There's createImage and createBackup. Both of them snapshot under the covers. | |
| 18:06:23 | sean-k-mooney | oh right | |
| 18:06:43 | sean-k-mooney | but createBackup uploads it as a snapshot to glance | |
| 18:07:06 | efried | point is, the image you get from createImage can be (non-awkwardly) used to clone your VM. Whereas createBackup is more tailored to rebuild. | |
| 18:07:07 | sean-k-mooney | does createimage upload as a snapshot i though it flattened the image | |
| 18:07:23 | sean-k-mooney | ok | |
| 18:07:56 | efried | rebuild is your same instance. Otherwise you're creating a new instance. If you're creating a new instance, it will get a new (or no) vTPM. | |
| 18:08:41 | efried | so the only quirk is that in order for rebuild to restore your vTPM, you have to use an image from createBackup. | |
| 18:08:43 | efried | which makes sense | |
| 18:08:59 | efried | if you use a random image from createImage, why would you expect your vTPM from your original instance to be restored? | |
| 18:09:23 | efried | conversely, if you're creating a brand new instance (from *any* image), why would you expect anything other than a fresh vTPM? | |
| 18:09:56 | sean-k-mooney | ya so i said previously that the vtpm should be tied to the lifetime of the instace so i think that makes sense | |
| 18:10:19 | efried | we've got an answer for all the move operations and shelve/unshelve. | |
| 18:10:35 | sean-k-mooney | we are explcitly saying the usecase of i create a vm snapshot and spawn 100 more is expresly out of scope (in that they wont all get the orginal vms vtpm) | |
| 18:11:08 | efried | I wouldn't phrase it as "out of scope". It's got clean, predictable behavior: you don't get the original VM's vTPM. | |
| 18:11:29 | sean-k-mooney | yes that is what i intened | |
| 18:11:37 | efried | which, for anyone using a vTPM, ought to make sense. "don't give my secure stuff to another instance" | |
| 18:11:48 | sean-k-mooney | copying the vtpm in that case is not supported intentionally | |
| 18:12:19 | efried | yeah, I'm just getting nitpicky about the language. "out of scope" and "not supported" imply "don't work". | |
| 18:12:28 | sean-k-mooney | so i would be ok with that set of constratits but im not really the person you have to convice | |
| 18:12:55 | eandersson | dansmith so we jumped from mitaka to rocky | |
| 18:13:09 | eandersson | So it's difficult to say exactly what part improved performance the most | |
| 18:13:17 | eandersson | but we believe placement played a big role here | |
| 18:13:35 | dansmith | eandersson: well, is it scheduling performance, instance listing performance, build time, etc? | |
| 18:13:40 | eandersson | scheduled is a lot better at least ~50% | |
| 18:13:49 | dansmith | sweet, so that's likely placement yeah | |
| 18:13:54 | eandersson | Also scaling computes | |
| 18:14:05 | dansmith | eandersson: what does "scaling computes" mean? | |
| 18:14:06 | eandersson | We initially limited ourself to 1000 computes per region | |
| 18:14:12 | eandersson | in Mitaka | |
| 18:14:23 | eandersson | but now believe we can reach a much higher number than that | |
| 18:14:55 | dansmith | eandersson: is that via cells or just in general? because I don't think we've done anything other than get *more* chatty to the db/mq since mitaka :) | |
| 18:14:57 | eandersson | And even if we do hit limitations we can now use cells | |
| 18:15:23 | eandersson | scheduler used to be really heavy on rmq | |
| 18:15:29 | eandersson | in mitaka | |
| 18:15:53 | sean-k-mooney | eandersson: do you run sepperate rmq instacne per openstack service | |
| 18:16:01 | eandersson | We do not | |
| 18:16:24 | eandersson | but even at 1k computes we are barely putting any stress on rmq at the moment | |
| 18:16:32 | eandersson | Of course we are not using ceilometer | |
| 18:16:34 | sean-k-mooney | im surpirsed you are getting to 1000 nodes on one cluster with neutron and nova sharing it | |
| 18:16:48 | dansmith | eandersson: so you're thinking that lowered load on rabbit from the scheduler lets you have more computes? | |
| 18:16:52 | sean-k-mooney | ya not using ceilometer helps | |
| 18:17:02 | eandersson | dansmith I think it helps | |
| 18:17:09 | dansmith | ack | |
| 18:18:12 | eandersson | We were getting a lot of slow api calls in mitaka and they are all very consistent now | |
| 18:19:06 | eandersson | The only problems we have with nova now is getting our super custom scheduling logic to scale | |
| 18:19:07 | dansmith | that's what I'm interested in specifically, | |
| 18:19:17 | dansmith | but without knowing which calls those were I can't really attribute them to anything | |
| 18:19:31 | dansmith | (that == slow api calls) | |