| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-10 | |||
| 11:53:13 | sean-k-mooney | wht the other one | |
| 11:53:58 | kashyap | 'rtm' | |
| 11:54:02 | sean-k-mooney | ya i have both hle and rtm | |
| 11:58:20 | sean-k-mooney | skylake so it should be enabled by defult which it does | |
| 11:58:43 | sean-k-mooney | kashyap: i tought you said the upstream kernel disabled tsx? | |
| 11:59:09 | kashyap | sean-k-mooney: It did (like I said, but its effect depends on the processor); what kernel are you running? | |
| 11:59:13 | sean-k-mooney | im running 5.10.8 | |
| 11:59:17 | sean-k-mooney | from fedora 32 | |
| 11:59:25 | kashyap | sean-k-mooney: Yours is older | |
| 11:59:30 | kashyap | sean-k-mooney: You need v5.11 or above | |
| 11:59:41 | kashyap | I even mentioned it in today's agenda meeting notes :D | |
| 12:00:04 | sean-k-mooney | so you ment it was disabled in an unreleased kernel | |
| 12:00:12 | sean-k-mooney | 5.11 is not released yet | |
| 12:00:35 | kashyap | sean-k-mooney: Wait, it was disabled in Oct 2019 | |
| 12:00:43 | kashyap | There have been plenty kernels released since then: https://github.com/torvalds/linux/commit/95c5824f75f3 | |
| 12:00:59 | sean-k-mooney | well fedora has it enabled | |
| 12:01:43 | sean-k-mooney | and the kernel config has CONFIG_X86_INTEL_TSX_MODE_OFF=y | |
| 12:02:07 | sean-k-mooney | i guess this is what you ment by you cant trust /proc/cpuinfo | |
| 12:02:17 | sean-k-mooney | let me check in a vm | |
| 12:02:19 | kashyap | sean-k-mooney: How are you saying it is "enabled"? If you're looking in /proc/cpuinfo, it is not | |
| 12:02:31 | kashyap | s/it is not/don't trust that/ :) | |
| 12:02:46 | kashyap | sean-k-mooney: Yes, as you see, your config has it _OFF=y | |
| 12:03:30 | sean-k-mooney | kashyap: that is the only interface we really have in userspace without poking registers | |
| 12:03:39 | sean-k-mooney | if we cant trust that we cant trust libvirt | |
| 12:04:51 | kashyap | sean-k-mooney: Wait. libvirt developers are telling me to use 'virsh domcapabilities' to detect what effect the 'tsx=off' parameter has | |
| 12:05:11 | kashyap | sean-k-mooney: The only other option is to poke for the MSR -- which I have to look, even if it is possible | |
| 12:06:28 | kashyap | sean-k-mooney: BTW, this not showing up "/proc/cpuinfo" is nothing new -- another flag is "mds-no" -- it also won't show up in /proc/cpuinfo | |
| 12:06:34 | sean-k-mooney | cpuid show RTM: restricted transactional memory = true | |
| 12:06:36 | kashyap | I wrote the upstream QEMU docs for it; /me looks | |
| 12:07:50 | kashyap | FWIW: https://lists.nongnu.org/archive/html/qemu-devel/2020-02/msg07164.html (qemu-cpu-models.rst: Document -noTSX, mds-no, taa-no, and tsx-ctrl) | |
| 12:08:20 | kashyap | sean-k-mooney: So you had it off in the kernel; but the hardware supports it. But `cpuid` shows RTM true? | |
| 12:08:59 | sean-k-mooney | yes cpu id shows true | |
| 12:09:15 | sean-k-mooney | and its off in the kernel | |
| 12:11:12 | sean-k-mooney | http://paste.openstack.org/show/802511/ | |
| 12:11:31 | sean-k-mooney | Skylake-Client-IBRS has it enabled | |
| 12:15:03 | kashyap | Yes, libvirt/QEMU introduced: "Skylake-Client-IBRS-noTSX" | |
| 12:20:02 | sean-k-mooney | ya so right now i do have rtm and hle in a vm on my laptop | |
| 12:21:28 | sean-k-mooney | with libvirt 6.1.0 | |
| 13:03:51 | sean-k-mooney | stephenfin: requirements_cehcks is unhappy but everything else passed https://review.opendev.org/c/openstack/os-vif/+/774694 | |
| 13:12:53 | openstackgerrit | Merged openstack/nova master: db: Compact Queens database migrations https://review.opendev.org/c/openstack/nova/+/759088 | |
| 14:41:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Use neutronclient's port binding APIs https://review.opendev.org/c/openstack/nova/+/706295 | |
| 14:45:29 | openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: Add support for microversion v2.88 https://review.opendev.org/c/openstack/python-novaclient/+/770573 | |
| 15:08:17 | sean-k-mooney | gibi: you can see nova security bug correct. | |
| 15:09:26 | sean-k-mooney | is there a group i can check i want to see who is on the list | |
| 15:09:28 | gibi | sean-k-mooney: hm, I saw some before, but then I was explicitly included in them. I think I'm not in the secu group in lauchpad | |
| 15:10:12 | sean-k-mooney | ah ok im being cauios with a bug im filing so im going to mark it as a security one initally but id like your input | |
| 15:10:20 | sean-k-mooney | ill add you to cc directly just in case | |
| 15:10:40 | gibi | sean-k-mooney: yes that will work | |
| 15:22:48 | openstackgerrit | Stephen Finucane proposed openstack/placement master: Move policy deprecation to base rules https://review.opendev.org/c/openstack/placement/+/772784 | |
| 15:23:14 | stephenfin | gmann: I found/fixed the bug on that and am +2 on it now ^ | |
| 15:24:58 | stephenfin | lyarwood: When you're back, would appreciate you weighing in on https://review.opendev.org/c/openstack/nova/+/751368 (stable patch) | |
| 15:26:01 | gmann | stephenfin: ah i see. thanks. | |
| 15:26:09 | stephenfin | it was so close to be perfect :D | |
| 15:26:13 | stephenfin | *being | |
| 15:27:34 | gmann | I thought we have broken something :) | |
| 15:32:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: cmd: Remove 'nova-manage db ironic_flavor_migration' https://review.opendev.org/c/openstack/nova/+/773776 | |
| 15:33:10 | stephenfin | gibi: You were +2 on that previously but I'd to address a doc failure. Care to revisit? Only change is to 'doc/source/cli/nova-manage.rst' ^ | |
| 15:33:20 | gibi | sure | |
| 15:33:24 | stephenfin | thanks | |
| 15:34:09 | stephenfin | gibi: We're _almost_ done with merging the main DB migration squash series (about time). We can probably look at removing that tweak to tox.ini once it's done, I guess? | |
| 15:35:00 | gibi | stephenfin: yepp, I can propose the rewert on top of the db series | |
| 15:35:10 | stephenfin | cool | |
| 15:44:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Liberty database migrations https://review.opendev.org/c/openstack/nova/+/759399 | |
| 15:44:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Add manage.py script https://review.opendev.org/c/openstack/nova/+/771419 | |
| 15:44:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Newton database migrations https://review.opendev.org/c/openstack/nova/+/759401 | |
| 15:44:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Mitaka database migrations https://review.opendev.org/c/openstack/nova/+/759400 | |
| 15:44:17 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Ocata database migrations https://review.opendev.org/c/openstack/nova/+/759402 | |
| 15:44:24 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Pike database migrations https://review.opendev.org/c/openstack/nova/+/759403 | |
| 15:44:26 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Queens database migrations https://review.opendev.org/c/openstack/nova/+/759404 | |
| 15:44:29 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Rocky database migrations https://review.opendev.org/c/openstack/nova/+/759405 | |
| 15:44:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Stein database migrations https://review.opendev.org/c/openstack/nova/+/759406 | |
| 15:44:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: apidb: Compact Train database migrations https://review.opendev.org/c/openstack/nova/+/771420 | |
| 15:44:55 | stephenfin | lyarwood, bauzas, melwitt: Can one of you take a look at https://review.opendev.org/c/openstack/nova/+/773727/ and https://review.opendev.org/c/openstack/nova/+/773728/ to fix the PPC64 gate, please | |
| 15:47:11 | bauzas | stephenfin: kinda try | |
| 15:48:04 | openstackgerrit | Stephen Finucane proposed openstack/os-vif master: Resolve dependency issues https://review.opendev.org/c/openstack/os-vif/+/774694 | |
| 15:48:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Revert "Run the db migration tests in the same test worker" https://review.opendev.org/c/openstack/nova/+/774889 | |
| 15:49:06 | gibi | stephenfin: here you are ^^ | |
| 15:49:11 | stephenfin | +2 | |
| 15:50:08 | gibi | I would keep the elastic signature still in place to see if the compaction really give us the speed bump on the gate I measured locally | |
| 15:53:48 | bauzas | stephenfin: https://review.opendev.org/c/openstack/nova/+/773728/ isn't needed for fixing the ppc64 gate job, righrt? | |
| 15:54:13 | stephenfin | bauzas: Yes, that's the crucial bit in fact | |
| 15:54:24 | bauzas | ok, then I misunderstood it | |
| 15:54:27 | bauzas | will review it | |
| 15:54:33 | stephenfin | http://lists.openstack.org/pipermail/openstack-discuss/2021-January/020088.html | |
| 15:55:20 | stephenfin | tl;dr: I added a patch to disable the USB controller if there were no USB devices in the XML, however, libvirt adds a USB keyboard and mouse automatically for guests on PPC64 hosts | |
| 15:55:40 | stephenfin | so it now fails | |
| 15:55:57 | bauzas | aha I see | |
| 15:56:16 | bauzas | it wants to add both the kb and the mouse but there are no longer usb controllers | |
| 15:56:21 | stephenfin | yup | |
| 15:56:26 | bauzas | gotcha, will look | |
| 15:56:41 | stephenfin | so we skip that logic on PPC64 now since we know we'll always need a controller on that platform | |
| 15:59:12 | lyarwood | stephenfin: sorry wasn't watching irc, I'll take a look at that and the other stuff from earlier after our next call | |
| 16:32:24 | dansmith | gibi: that's nifty, I didn't know about that grouping thing | |
| 16:33:47 | dansmith | gibi: did you happen to notice if the IO-related timeouts were specific to one provider? | |
| 16:34:20 | dansmith | tripleo was having job timeouts on one of their jobs that did a lot of IO and vexxhost switched our flavor to double the IOPs | |
| 16:35:26 | gibi | dansmith: the failures was spread amongst providers. at least I haven't noticed a pattern | |
| 16:35:32 | dansmith | okay | |
| 16:35:59 | gibi | I assumed it was IO limited as I was able to reproduce the extreme slowness locally if I IO deprived the machine | |
| 16:36:14 | dansmith | my parallel devstack ran all of those migrations in parallel for a very nice speedup on local hardware, | |
| 16:36:32 | dansmith | so maybe with 75% faster due to compaction we'll get a nice bump in the io-limited workers too | |