| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-14 | |||
| 14:47:42 | sean-k-mooney | which the cpu feature flags are in another vail | |
| 14:47:54 | dansmith | "is broken" or "was broken" or "has a software fix" are terrible traits to me | |
| 14:48:02 | sean-k-mooney | dansmith: ya i dont like tracking vulnerablity in placement period | |
| 14:48:07 | dansmith | sean-k-mooney: agreen | |
| 14:48:09 | dansmith | *agreed | |
| 14:48:30 | dansmith | I also don't think that tracking cpu flags in placement for non-feature flags makes sense | |
| 14:48:57 | dansmith | AVX2, yes... flags that just imply that a microcode fix has been applied, not so much | |
| 14:49:01 | sean-k-mooney | i stil am concerned that adding any of the security realted flag give me a vector to upload an image with a require/forbidn trait and target a vulnerable host | |
| 14:49:16 | dansmith | sean-k-mooney: for serious | |
| 14:49:17 | kashyap | dansmith: How about this idea (that I discussed here on PS:4 as "Another Idea" - https://review.opendev.org/#/c/655193/): | |
| 14:49:20 | kashyap | Make Nova check the 'sysfs' (/sys/devices/system/cpu/vulnerabilities) directory for vulnerabilities. And e.g. if it reports "Vulnerable" (instead of "Mitigation") for Meltdown (or other flaws), print a warning that the host is vulnerable, and on next release refuse to start the VMs | |
| 14:49:46 | sean-k-mooney | dansmith: :) | |
| 14:50:16 | dansmith | kashyap: I have a hard time understanding why this is a nova thing at all.. configs to turn on compatibility as needed makes sense, since we are controlling the definition of the VMs, | |
| 14:50:20 | sean-k-mooney | kashyap: i dont think its novas place to do that | |
| 14:50:28 | dansmith | but the management of the host-level patching is not our deal, IMHO | |
| 14:50:33 | dansmith | what sean-k-mooney said | |
| 14:51:01 | kashyap | dansmith: Nah, we don't _patch_ it. But isn't it fair game for Nova to tell that: "hey, your host is vulnerable to critical flaws, launching VMs there is dangerous?" | |
| 14:51:07 | dansmith | same reason we don't refuse to boot instances because we're concerned that our cpu fans are spinning too slowly | |
| 14:51:10 | dansmith | kashyap: not IMHO | |
| 14:51:13 | kashyap | I see. | |
| 14:51:25 | kashyap | Just wondering out loud. My umbilical cord isn't tied to that idea :-) | |
| 14:52:47 | sean-k-mooney | kashyap: depending on your enviornment (a air gapped secure datacenter) you ligitamately may want to have a vulnerable system for the performace improvments. i agree with dansmith that it should not be novas role to dictate your securety and threat model | |
| 14:53:00 | kashyap | One would think it's reasonable for Nova to prompt admins to secure their hypervisors. | |
| 14:53:11 | dansmith | sean-k-mooney: yeah, definitely agree.. I have unpatched systems for reasons :) | |
| 14:53:18 | kashyap | sean-k-mooney: Yeah, I've considered that point -- on someone _intentionally_ wanting to run unpatched | |
| 14:53:24 | kashyap | ... precisely for the perf reasons. | |
| 14:53:47 | sean-k-mooney | kashyap: you could make the same argument for libvirt if you go that route or qemu | |
| 14:54:10 | dansmith | or fan speed, or cpu temperature, or firmware patch levels, or os patch levels, or .... | |
| 14:54:21 | kashyap | Well, that doesn't make sense in the _common_ case for those components. Not everyone is sitting there with "air-gapped secure datacenters". | |
| 14:54:26 | sean-k-mooney | the kernel warns the user and i think that is where the warning should be | |
| 14:56:28 | kashyap | sean-k-mooney: Are you referring to the 'sysfs' directory as "kernel warns"? | |
| 14:57:01 | sean-k-mooney | kashyap: well i ment the message in dmesg form the early boot but sysfs can be used later at runtime | |
| 14:57:23 | sean-k-mooney | its also show in thing like lscpu | |
| 15:01:30 | sean-k-mooney | actully its not in lscpu but is in /proc/cpuinfo in the bugs line "bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf" | |
| 15:01:56 | kashyap | sean-k-mooney: Yeah, it's _not_ in `lscpu` | |
| 15:02:14 | sean-k-mooney | ya it would be a nice enhancement to add it | |
| 15:02:22 | kashyap | Right, it's in /proc/cpuinfo. And of course, 'sysfs' is a better place. | |
| 15:04:33 | sean-k-mooney | yep and you allso get entires in dmesg as the are detected on kenel boot with the mitigations that are applied | |
| 15:04:36 | sean-k-mooney | like this | |
| 15:04:39 | sean-k-mooney | 0.033797] Spectre V2 : Mitigation: Full generic retpoline | |
| 15:04:41 | sean-k-mooney | [ 0.033797] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch | |
| 15:04:43 | sean-k-mooney | [ 0.033798] Spectre V2 : Spectre v2 mitigation: Enabling Indirect Branch Prediction Barrier | |
| 15:04:45 | sean-k-mooney | [ 0.033798] Spectre V2 : Enabling Restricted Speculation for firmware calls | |
| 15:04:47 | sean-k-mooney | [ 0.033799] Spectre V2 : Spectre v2 cross-process SMT mitigation: Enabling STIBP | |
| 15:04:49 | sean-k-mooney | [ 0.033800] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp | |
| 15:05:19 | sean-k-mooney | which is the same info that gets stroed in cat /sys/devices/system/cpu/vulnerabilities/* | |
| 15:05:55 | openstackgerrit | Gorka Eguileor proposed openstack/nova master: Use os-brick locking for volume attach and detach https://review.opendev.org/614190 | |
| 15:13:26 | mriedem | bauzas: dansmith: can one of you hit this stable/stein test-only backport to keep things moving? https://review.opendev.org/#/c/658929/ | |
| 15:20:25 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Stop handling 'update_cells' on 'BandwidthUsage.create' https://review.opendev.org/651303 | |
| 15:20:25 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'instance_update_from_api' https://review.opendev.org/651302 | |
| 15:20:26 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove cells code https://review.opendev.org/651306 | |
| 15:20:26 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Stop handling cells v1 for instance naming https://review.opendev.org/651304 | |
| 15:20:27 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove conductor_api and _last_host_check from manager.py https://review.opendev.org/651059 | |
| 15:30:42 | openstackgerrit | Adrian Chiris proposed openstack/nova master: [FUP] Follow-up patch for SR-IOV live migration https://review.opendev.org/659101 | |
| 15:51:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add an online migration for PciDevice.uuid https://review.opendev.org/530905 | |
| 15:51:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Modify PciDevice.uuid generation code https://review.opendev.org/530487 | |
| 15:52:53 | stephenfin | dansmith: If you have any free time today, could you take a look at those two patches again? ^ They're the sixth and seventh oldest ones I have, respectively :) | |
| 15:56:08 | dansmith | stephenfin: is there something that is going to need this? | |
| 15:56:19 | dansmith | not that it's a bad thing to have proper uuids, but... just wondering | |
| 15:56:53 | stephenfin | We have a chunk of legacy code that is present only because we can't be sure we don't have UUIDs. I'd like to remove that | |
| 15:56:57 | stephenfin | So it's tech debt | |
| 15:58:10 | dansmith | okay | |
| 16:00:11 | mriedem | stephenfin: the bottom one is so obviously copied from the bdm code it still mentions bdms in it | |
| 16:00:46 | stephenfin | mriedem: Yuuup (though the reference is obv a mistake) | |
| 16:01:30 | stephenfin | mdbooth and I discussed dragging it out into something standardized a long time back, but we figured it would be better wait for at least one more user before proceeding down the generic util path | |
| 16:04:14 | openstackgerrit | Merged openstack/python-novaclient master: [Docs] Update client docs to add reason and locked options https://review.opendev.org/659000 | |
| 16:10:33 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add code comment to _instance_update https://review.opendev.org/659112 | |
| 16:28:03 | openstackgerrit | Adrian Chiris proposed openstack/nova master: [FUP] Follow-up patch for SR-IOV live migration https://review.opendev.org/659101 | |
| 16:36:28 | logan- | hello, in OSA we've got a failure bumping our nova testing SHA (https://review.opendev.org/#/c/658208/) due to an api_db sync failure: | |
| 16:36:30 | logan- | http://logs.openstack.org/08/658208/3/check/openstack-ansible-deploy-aio_metal-ubuntu-bionic/6c21b9f/logs/ara-report/result/2b065096-d2f5-497b-a29b-2833502c0459/ | |
| 16:36:41 | logan- | "AttributeError: 'module' object has no attribute 'COMPUTE_IMAGE_TYPE_AKI'" | |
| 16:36:49 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Add 'flavor-extra-spec-image-property-validation-extended' spec https://review.opendev.org/638734 | |
| 16:36:59 | mriedem | logan-: which version of os-traits? | |
| 16:37:45 | mriedem | you need os-traits>=0.12.0 | |
| 16:37:51 | logan- | checking | |
| 16:37:53 | logan- | ok thanks | |
| 16:38:14 | dtantsur | hey folks, we see this one rocky in ironic: http://logs.openstack.org/71/658771/4/check/ipa-tempest-dsvm-partition-bios-ipmi-iscsi-tinyipa-src/fb9d0bb/logs/screen-n-cpu.txt.gz?level=INFO | |
| 16:38:36 | dtantsur | can it be https://github.com/openstack/nova/commit/35bda4ec385e4c2b3d4cee07467f5077b13b1dd9 ? | |
| 16:38:49 | mriedem | dtantsur: which thing? the StrictVersion? yes. | |
| 16:38:57 | mriedem | er wait | |
| 16:39:22 | mriedem | the StrictVersion thing has been around awhile https://bugs.launchpad.net/nova/+bug/1793766 | |
| 16:39:23 | openstack | Launchpad bug 1793766 in OpenStack Compute (nova) "An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: StrictVersion instance has no attribute 'version'" [Low,Confirmed] | |
| 16:39:29 | mriedem | but maybe it's hitting more now because of ^ | |
| 16:39:51 | dtantsur | I'm not sure if it's on nova or ironicclient side tbh | |
| 16:40:22 | logan- | thanks for the tip mriedem, it looks like we're pulling in 0.11.0 somehow. http://logs.openstack.org/08/658208/3/check/openstack-ansible-deploy-aio_metal-ubuntu-bionic/6c21b9f/logs/ara-report/result/7f027c76-34d4-4dfa-a06a-d91722043b58/ | |
| 16:40:36 | dtantsur | mriedem: then we end up with http://logs.openstack.org/71/658771/4/check/ipa-tempest-dsvm-partition-bios-ipmi-iscsi-tinyipa-src/fb9d0bb/logs/screen-n-cpu.txt.gz#_May_14_12_21_18_271733 | |
| 16:40:45 | dtantsur | I'm not sure if the StrictVersion thingy is the cause, but looks so | |
| 16:40:57 | logan- | oh we need to bump our requirements sha | |
| 16:40:58 | logan- | heh | |
| 16:41:01 | mriedem | logan-: stale upper-constraints? | |
| 16:41:03 | mriedem | oh ok | |
| 16:41:35 | logan- | yup, that should fix it up. thanks! | |
| 16:41:45 | stephenfin | sean-k-mooney: If you're around for a while longer, you might fancy looking at https://review.opendev.org/638734 again. Think I've addressed most of your concerns (by focussing solely on flavor extra specs) | |
| 16:46:25 | sean-k-mooney | oh the validation spec sure ill take a look | |
| 16:47:42 | openstackgerrit | Merged openstack/nova stable/stein: Add functional confirm_migration_error test https://review.opendev.org/658929 | |
| 16:53:15 | tssurya | mriedem: could you have a look when you have time ? https://review.opendev.org/#/c/658110/ - thanks in advance | |
| 17:11:15 | sean-k-mooney | stephenfin: adrianc i left some comments on https://review.opendev.org/#/c/659101/2 but this actully intduces more nits then it fixes at least from my view point | |
| 17:54:06 | sean-k-mooney | stephenfin: done. i think i have a better solution that i more generic then what you proposed but still support a yaml file for operators to customise | |
| 17:55:21 | sean-k-mooney | all standard extra specs however would be validated in python and we woudl have a stevedor entry point so that cyborg, rmd,os-traits extra specs could also be validated | |
| 18:48:48 | adrianc | sean-k-mooney: Thanks, ill take a look tommorow ? | |