| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-19 | |||
| 16:06:06 | stephenfin | jaypipes: For instances with dedicated CPUs, yes. However, we need to avoid breaking users | |
| 16:06:13 | stephenfin | So that knob would have to default to on | |
| 16:06:58 | stephenfin | The reason I want that is because a hyperthread != a core and I think the decision to model it as one was a mistake on day one | |
| 16:06:59 | jaypipes | stephenfin: ack. no disagreement from me on that | |
| 16:07:12 | jaypipes | stephenfin: agreed. | |
| 16:10:21 | openstackgerrit | sean mooney proposed openstack/nova-specs master: [WIP] generic device discovery policy https://review.openstack.org/603805 | |
| 16:20:37 | openstackgerrit | Merged openstack/nova stable/queens: Fix DB archiver AttributeError due to wrong table name attribute used https://review.openstack.org/599882 | |
| 16:20:44 | openstackgerrit | Merged openstack/nova stable/queens: VMware: fix TypeError while get console log https://review.openstack.org/591365 | |
| 16:20:51 | openstackgerrit | Merged openstack/nova stable/queens: Move conductor wait_until_ready() delay before manager init https://review.openstack.org/599200 | |
| 16:28:59 | mriedem | dansmith: i'm good to skip; plan on starting my poc for cross-cell resize today | |
| 16:29:11 | dansmith | oh boy | |
| 16:30:33 | openstackgerrit | Merged openstack/nova stable/ocata: Return 400 when compute host is not found https://review.openstack.org/590649 | |
| 16:31:38 | stephenfin | jaypipes: If you don't have one already, a placement cheatsheet (in PDF and therefore printable) would be super useful | |
| 16:31:58 | jaypipes | stephenfin: also, I agree with you about getting rid of emulator_threads_policy eventually, but is it 100% critical that I update the cpu-resource-tracking spec with information about that? | |
| 16:32:24 | jaypipes | stephenfin: you mean cheatsheet about the REST API? | |
| 16:32:29 | jaypipes | stephenfin: or something else? | |
| 16:32:41 | jaypipes | stephenfin: or you mean a glossary of placement terms? | |
| 16:33:26 | stephenfin | jaypipes: No, it can be be handled separately. However, I imagine you do need to take cpu overhead, which is incremented for each dedicated emulator thread core, into account | |
| 16:33:31 | openstackgerrit | Jack Ding proposed openstack/nova master: Correct instance port binding for rebuilds/reboots https://review.openstack.org/603844 | |
| 16:33:55 | stephenfin | jaypipes: Both. Glossary of terms followed by examples of either the REST API or using osc-placement | |
| 16:34:17 | jaypipes | stephenfin: ack | |
| 16:34:21 | stephenfin | jaypipes: Just an idea, obviously. I might even start drafting something myself some point this week | |
| 16:34:44 | jaypipes | stephenfin: cool. let me know when you get to the "what is a consumer?" part... | |
| 16:36:43 | stephenfin | jaypipes: FYI, the reason I brought up the whole "let's kill 'isolate' idea" was that that approach seemed easier than handling the CPU overhead thing *and* it reduced complexity for the user in the process, which feels like one of the goals | |
| 16:37:26 | stephenfin | But again, not a blocker :) | |
| 16:39:07 | jaypipes | stephenfin: yes, agreed. glad it's not a blocker though. | |
| 16:39:27 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: libvirt: mdevs returning parent and vendor PCI info https://review.openstack.org/562304 | |
| 16:40:21 | bauzas | jaypipes: stephenfin: can one of you reapprove https://review.openstack.org/562304 ? I just rebased it out of the existing series and I need it for the reshaper change | |
| 16:40:52 | bauzas | I'm surprised I lost my +2s on the rebase but meh | |
| 16:41:02 | bauzas | probably because I changed the commit id | |
| 16:41:15 | bauzas | oh yeah, I changed the message, hence why | |
| 16:41:30 | jaypipes | stephenfin: feel free to re-+W bauzas patch | |
| 16:41:44 | bauzas | jaypipes: thanks | |
| 16:42:15 | bauzas | I need the parenting relationship for reshaping the VGPU resources to the right PGPU RP | |
| 16:42:54 | stephenfin | bauzas: Done | |
| 16:43:15 | bauzas | stephenfin: ack, thanks | |
| 17:01:41 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | |
| 17:09:08 | tobias-urdin | ping on a possible upgrade path bug q->r https://bugs.launchpad.net/nova/+bug/1793353 | |
| 17:09:09 | openstack | Launchpad bug 1793353 in OpenStack Compute (nova) "broken upgrade path q->r requirement for oslo.db" [Undecided,New] | |
| 17:37:11 | openstackgerrit | David Rabel proposed openstack/nova master: Really use source image format as default for snapshot_image_format https://review.openstack.org/603855 | |
| 17:51:23 | jiteka | hello, could someone confirm that migrate feature is working with flavor using vCPU pinning and Numa in Mitaka ? | |
| 18:01:58 | openstackgerrit | Merged openstack/nova-specs master: Propose configurable maximum number of volumes to attach https://review.openstack.org/597306 | |
| 18:10:49 | mriedem | jiteka: cfriesen might know | |
| 18:11:09 | mriedem | there were some patches that mellanox and windriver worked on to make that work, but i can't remember which release in which those patches landed | |
| 18:11:28 | mriedem | jiteka: cold migrate i mean, not live migration | |
| 18:11:36 | mriedem | numa/pinned cpus does not work with live migration | |
| 18:11:53 | mriedem | https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/numa-aware-live-migration.html | |
| 18:28:40 | openstackgerrit | Merged openstack/nova stable/ocata: Add recreate test for RT.stats bug 1784705 https://review.openstack.org/588076 | |
| 18:28:40 | openstack | bug 1784705 in OpenStack Compute (nova) ocata "ResourceTracker.stats can leak across multiple ironic nodes" [High,In progress] https://launchpad.net/bugs/1784705 - Assigned to Matt Riedemann (mriedem) | |
| 18:31:27 | mriedem | dansmith: would be good to get this pike backport in https://review.openstack.org/#/c/599883/ | |
| 18:32:14 | dansmith | mriedem: hmm, did I do that? ISTR some reasoning there.. maybe sqla version? | |
| 18:32:59 | mriedem | it was originally your code yes, but table.name is standard | |
| 18:44:04 | mriedem | dansmith: does anything jump out at you for the source of the IndexError in this failure? http://logs.openstack.org/72/600372/1/gate/openstack-tox-py35/dcfd363/job-output.txt.gz | |
| 18:44:56 | mriedem | looks like something between oslo.db/sqla/pymysql/eventlet switches context and we timeout after some huge amount of time | |
| 18:45:07 | mriedem | nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_models_sync [664.512994s] ... FAILED | |
| 18:45:11 | mriedem | nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_models_sync [39.644814s] ... ok | |
| 18:45:30 | mriedem | clearly we're not just missing some short timeout window | |
| 18:46:09 | dansmith | still waiting for the damn log to load | |
| 18:48:14 | dansmith | wow | |
| 18:48:54 | mriedem | yeah we have a lot of these failures in the gate across master, rocky and queens, | |
| 18:48:55 | mriedem | https://bugs.launchpad.net/cinder/+bug/1793364 | |
| 18:48:55 | openstack | Launchpad bug 1793364 in OpenStack Compute (nova) "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [High,Confirmed] | |
| 18:49:05 | mriedem | nova and cinder b/c they are using the same test fixtures | |
| 18:49:55 | dansmith | hmm | |
| 18:50:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Add contributor guide for upgrade status checks https://review.openstack.org/596902 | |
| 18:50:39 | mriedem | maybe "connection.scalar(select([1]))"? | |
| 18:50:42 | dansmith | does the index complaint surprise you? because it looks like a nonsense name | |
| 18:51:08 | mriedem | well i see this too | |
| 18:51:09 | mriedem | sqlalchemy.exc.ResourceClosedError: This result object does not return rows. It has been closed automatically | |
| 18:51:27 | dansmith | yeah | |
| 18:52:34 | mriedem | http://status.openstack.org/elastic-recheck/data/integrated_gate.html Overall Categorization Rate: 15.4% | |
| 18:52:47 | mriedem | meaning we (openstack) have a shit load of uncategorized failures killing stuff in the gate | |
| 18:52:55 | mriedem | which is why it's taking us days to merge code | |
| 18:57:32 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%20%20connection.scalar(select(%5B1%5D))'%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 19:04:33 | cfriesen | jiteka: I think cold migration should work in mitaka. I think there are some gotchas where resource tracking isn't accurate until the next audit, and maybe there are some resource tracking issues if you revert a resize. | |
| 19:06:13 | tssurya | dansmith: ack, let's skip | |
| 19:06:16 | cfriesen | jiteka: a bit more digging shows that PCI devices were not cold migratable until Newton. | |
| 19:07:13 | mriedem | cfriesen: did you find a specific change that makes that work? | |
| 19:07:42 | mriedem | b/c we could make a note of that on https://docs.openstack.org/nova/latest/admin/pci-passthrough.html | |
| 19:09:34 | cfriesen | mriedem: 5129f48226c I think | |
| 19:10:30 | cfriesen | actually, looks like there are multiple: https://bugs.launchpad.net/nova/+bug/1512880 | |
| 19:10:31 | openstack | Launchpad bug 1512880 in OpenStack Compute (nova) newton "Failed cold migration with SR-IOV" [Medium,Fix released] | |
| 19:10:39 | mriedem | yeah was just going to say that https://review.openstack.org/#/q/topic:bug/1512880+(status:open+OR+status:merged) | |
| 19:11:32 | mriedem | i think it's fair to say it didn't work until newton | |
| 19:28:17 | lucidguy | I believe this bug applies to me, unfortunately I don't know how to resolve, assistance? https://bugs.launchpad.net/tripleo/+bug/1785568 | |
| 19:28:17 | openstack | Launchpad bug 1785568 in OpenStack Compute (nova) "Multiple migration requests for same vm might fail" [Undecided,Incomplete] | |
| 19:29:19 | mriedem | cfriesen: so if you have a minute, could you just update the note at the top of https://docs.openstack.org/nova/latest/admin/pci-passthrough.html about sriov to also mention that cold migration of servers with sriov ports attached didn't work until newton and reference that bug? | |
| 19:42:52 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | |
| 19:46:23 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | |
| 20:12:14 | openstackgerrit | Merged openstack/nova-specs master: fix spelling mistake https://review.openstack.org/573479 | |
| 20:32:25 | cfriesen | anyone know anything about "bad magic number in nova.conf.crypto" when trying to build nova docs? | |
| 20:33:34 | cfriesen | http://paste.openstack.org/show/730383/ | |
| 20:35:25 | mriedem | i've seen that i think... | |
| 20:37:02 | mriedem | py34 isn't supported... | |
| 20:37:06 | mriedem | not sure if that would be related | |
| 20:37:53 | mriedem | also, | |
| 20:37:57 | mriedem | nova.conf.crypto isn't in master | |
| 20:38:05 | mriedem | is that a starlingx special? | |
| 20:38:29 | mriedem | oh pike | |
| 20:38:29 | mriedem | https://github.com/openstack/nova/blob/stable/pike/nova/conf/crypto.py | |