Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-03
15:17:02 ygk_12345 sean-k-mooney we have a another similar setup, there also I observer broke pipe errors but the console is functioning fine there
15:17:58 ygk_12345 sean-k-mooney do u thin k the problem is with the netwrok issues ?
15:19:06 sean-k-mooney dansmith: by the way after our conversation with sundar yesterday i decied to look at option of emulating pci device in the kernel again. we may be able ot use the PCI Endpoint Framework https://www.kernel.org/doc/html/latest/PCI/endpoint/index.html to create device we could use for pci passthough testing and cyborg testing
15:19:28 dansmith sean-k-mooney: sweet
15:20:04 sean-k-mooney i need to play around with it as the test driver is not complied in to the ubunut kernel module extra so il need to see how to compile it but ill let you know how it goes
15:20:29 dansmith cool
15:20:42 sean-k-mooney it should allow use to create pci devices by creating folders with in /sys
15:21:22 sean-k-mooney https://www.kernel.org/doc/html/latest/PCI/endpoint/pci-test-howto.html#creating-pci-epf-test-device
15:22:09 sean-k-mooney if it works we shoudl be able to set the vendor id and prodcut id ot like 1234:42 then assert that a pci deivce with that vendor and prodcit id is availabel and passed to the guest
15:25:30 stephenfin mriedem: Dumb question, but the 'device_id' in a neutron port response will always be the instance's UUID, not the ID, right?
15:28:11 sean-k-mooney stephenfin: yes it will never be the db short id
15:28:19 stephenfin ta
15:30:55 dansmith stephenfin: if it were the db id we wouldn't be able to distinguish one instance over another across cells
15:31:13 mriedem we also don't expose the server primary key id out of the rest api
15:31:57 sean-k-mooney impliying that if it was teh primary key id that neutron would not be able to use it in api queries
15:32:30 sean-k-mooney we did expose the hypervior primary key although i think that was by mistake.
15:32:43 mriedem it wasn't by mistake for hypervisors,
15:32:51 mriedem they originally didn't have uuids
15:32:57 sean-k-mooney ah ok
15:32:57 mriedem same with services and lots of other things
15:57:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Cache security group driver https://review.opendev.org/697122
16:26:31 dansmith anybody want to +W this so Sundar can just rebase on master? https://review.opendev.org/#/c/695985/
16:26:41 dansmith s/rebase/rebase the cyborg stuff/
16:27:09 mriedem gibi: i replied to your questions in https://review.opendev.org/#/c/637058/, thanks. i'll stack a change on top of the series to see what replacing that setup_networks_on_host hack would look like if we just implement port binding deleting in cleanup_instance_network_on_host
16:28:50 mriedem dansmith: looking
16:30:06 gibi mriedem: ack, thanks
16:46:24 openstackgerrit Thierry Carrez proposed openstack/nova master: Remove unused rootwrap filters https://review.opendev.org/697134
17:36:14 dansmith eandersson: what perf improvements in nova specifically have shifted your scale focus away from nova?
17:40:09 efried dansmith: Would you please have another look at the vTPM spec. I'd like to get your +2 before you bugger off til 2020. https://review.opendev.org/#/c/686804/
17:40:51 dansmith efried: I've skimmed it.. tbh, I'm really like -0.9 on it, so if you want me to review it, it's probably going to be not helpful to your effort
17:41:21 dansmith I'm not sure I see the benefit outweighing the nightmare of users trying to predict the behavior
17:41:46 dansmith I was thinking it was required for secure boot, but talking with folks I realize it's not, so ... it's just hard to justify I think
17:45:09 sean-k-mooney for what its worth the out of tree hyperv driver support vtpm for 3-4 years already
17:45:36 sean-k-mooney its part fo there sheilded vms thing but i have no idea how they solved any of the issues with move operations
17:45:49 efried did they?
17:46:21 sean-k-mooney yep ill get the link. did i not send this to you already
17:46:27 dansmith sean-k-mooney: given how hyperv moves things I expect it's tracked by the hypervisor and so it just works, but I could be wrong
17:46:54 sean-k-mooney https://github.com/openstack/compute-hyperv/commit/f37ce8b6bb0eb88a367239698ba7c3df3b64db38
17:47:01 dansmith the question is more about how things like snapshot works
17:47:10 sean-k-mooney they have an os_vtpm image property
17:50:00 sean-k-mooney they must have patches against nova because the ImageMetaProps object does not accept os_vtpm or os_shielded_vm
17:53:13 dansmith looks like maybe they're doing some mangling of instance.hostname to reference specific keys or something?
17:53:24 dansmith and they're illegally stashing their own stuff in insance.metadata
17:53:35 dansmith so all manner of hacks in that implementation
17:53:43 sean-k-mooney yep
17:53:55 sean-k-mooney and none of this is supported in the in tree one
17:56:42 sean-k-mooney im looking at there snapshot function now https://github.com/openstack/compute-hyperv/blob/cb203978f262f31790592b2a0692fc2acaaef33d/compute_hyperv/nova/snapshotops.py#L66 but i think its just of the image so they dont snapshot the tpm?
17:57:32 dansmith I dunno, but since they've already broken many rules of user interaction, looking further into how they handle it doesn't seem like it would help guide us
17:57:58 sean-k-mooney it proably wont but it looks like they just ignored it
17:58:27 sean-k-mooney so i guess there stance was the tpm state would not be resoreted if you restored form a snapshot
17:58:39 sean-k-mooney the same way non root disks are not restored
17:59:40 dansmith or there is magic in the hostname mangling stuff so that if you recreate an instance with the right magic hostname, it will re-get the tpm stored in the hypervisor?
18:00:39 dansmith given that they basically implement features by bounty in that out of tree driver, I expect they implemented just the part of the solution that would suffice for the one customer asking for it
18:01:26 sean-k-mooney i dont think they are manageling the path in the metatada but yes i would guess the custoemr did not ask for snapshotiing so they did not add it
18:02:54 efried To me, the only severe issue is evacuate. The rest should be easy to understand with proper documentation.
18:02:59 sean-k-mooney i think they were using the vtpm soly for disk encryption
18:03:19 efried "vTPM is tied to your instance. Don't expect to be able to snapshot and clone it."
18:04:13 efried "You can back up and restore as long as you use 'backup'."
18:04:44 sean-k-mooney ? you use backup
18:05:26 sean-k-mooney you mean as long as you rebuild form a snapshot and dont launch an new instance right
18:05:46 efried No. I mean as long as you use rebuild from an image created via the createBackup server action
18:06:07 sean-k-mooney right but does that not just create a shapshot
18:06:11 efried 'snapshot' isn't a server action. There's createImage and createBackup. Both of them snapshot under the covers.
18:06:23 sean-k-mooney oh right
18:06:43 sean-k-mooney but createBackup uploads it as a snapshot to glance
18:07:06 efried point is, the image you get from createImage can be (non-awkwardly) used to clone your VM. Whereas createBackup is more tailored to rebuild.
18:07:07 sean-k-mooney does createimage upload as a snapshot i though it flattened the image
18:07:23 sean-k-mooney ok
18:07:56 efried rebuild is your same instance. Otherwise you're creating a new instance. If you're creating a new instance, it will get a new (or no) vTPM.
18:08:41 efried so the only quirk is that in order for rebuild to restore your vTPM, you have to use an image from createBackup.
18:08:43 efried which makes sense
18:08:59 efried if you use a random image from createImage, why would you expect your vTPM from your original instance to be restored?
18:09:23 efried conversely, if you're creating a brand new instance (from *any* image), why would you expect anything other than a fresh vTPM?
18:09:56 sean-k-mooney ya so i said previously that the vtpm should be tied to the lifetime of the instace so i think that makes sense
18:10:19 efried we've got an answer for all the move operations and shelve/unshelve.
18:10:35 sean-k-mooney we are explcitly saying the usecase of i create a vm snapshot and spawn 100 more is expresly out of scope (in that they wont all get the orginal vms vtpm)
18:11:08 efried I wouldn't phrase it as "out of scope". It's got clean, predictable behavior: you don't get the original VM's vTPM.
18:11:29 sean-k-mooney yes that is what i intened
18:11:37 efried which, for anyone using a vTPM, ought to make sense. "don't give my secure stuff to another instance"
18:11:48 sean-k-mooney copying the vtpm in that case is not supported intentionally
18:12:19 efried yeah, I'm just getting nitpicky about the language. "out of scope" and "not supported" imply "don't work".
18:12:28 sean-k-mooney so i would be ok with that set of constratits but im not really the person you have to convice
18:12:55 eandersson dansmith so we jumped from mitaka to rocky
18:13:09 eandersson So it's difficult to say exactly what part improved performance the most
18:13:17 eandersson but we believe placement played a big role here
18:13:35 dansmith eandersson: well, is it scheduling performance, instance listing performance, build time, etc?
18:13:40 eandersson scheduled is a lot better at least ~50%
18:13:49 dansmith sweet, so that's likely placement yeah
18:13:54 eandersson Also scaling computes
18:14:05 dansmith eandersson: what does "scaling computes" mean?
18:14:06 eandersson We initially limited ourself to 1000 computes per region
18:14:12 eandersson in Mitaka
18:14:23 eandersson but now believe we can reach a much higher number than that
18:14:55 dansmith eandersson: is that via cells or just in general? because I don't think we've done anything other than get *more* chatty to the db/mq since mitaka :)
18:14:57 eandersson And even if we do hit limitations we can now use cells
18:15:23 eandersson scheduler used to be really heavy on rmq
18:15:29 eandersson in mitaka
18:15:53 sean-k-mooney eandersson: do you run sepperate rmq instacne per openstack service
18:16:01 eandersson We do not
18:16:24 eandersson but even at 1k computes we are barely putting any stress on rmq at the moment

Earlier   Later